CN113499094B - Heart color Doppler ultrasound examination device and method guided by vision and force feedback - Google Patents
Heart color Doppler ultrasound examination device and method guided by vision and force feedback Download PDFInfo
- Publication number
- CN113499094B CN113499094B CN202110776269.9A CN202110776269A CN113499094B CN 113499094 B CN113499094 B CN 113499094B CN 202110776269 A CN202110776269 A CN 202110776269A CN 113499094 B CN113499094 B CN 113499094B
- Authority
- CN
- China
- Prior art keywords
- mechanical arm
- personal computer
- industrial personal
- force feedback
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/54—Control of the diagnostic device
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/44—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device
- A61B8/4444—Constructional features of the ultrasonic, sonic or infrasonic diagnostic device related to the probe
- A61B8/4461—Features of the scanning mechanism, e.g. for moving the transducer within the housing of the probe
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
The invention relates to a heart color Doppler ultrasound examination device and method guided by vision and force feedback. The mechanical arm and the force feedback remote controller are arranged in two independent rooms through communication between the first industrial personal computer and the second industrial personal computer, so that remote color Doppler ultrasound is realized, and contact between doctors and patients is avoided; the doctor controls the force feedback remote controller, and the control information is transmitted to the mechanical arm through the second industrial personal computer and the second industrial personal computer so as to control the movement of the mechanical arm; the mechanical arm transmits the information of the external force to the force feedback remote controller through the first industrial personal computer and the second industrial personal computer, a doctor can feel the effect of the force in real time on the force feedback remote controller, and in addition, the monitoring video is checked in real time through the display screen of the second industrial personal computer. There is no risk of any disease transmission between doctor and patient; and the force feedback remote control can convert the external force information received by the mechanical arm into the force feedback information received by the hands of the doctor in real time, thereby ensuring the accuracy of the doctor in remote control and improving the safety of color Doppler ultrasound examination.
Description
Technical Field
The invention belongs to the technical field of medical appliances, and particularly relates to a heart color Doppler ultrasound examination device and method guided by vision and force feedback.
Background
At present, color ultrasonic inspection is realized by a doctor operating a color ultrasonic probe to contact with a patient; this inspection method has the following problems: 1. contact between the patient and the doctor is unavoidable, and the doctor risks being infected when checking for patients with unknown infectious diseases; 2. the color ultrasonic examination needs a skilled doctor to obtain clear images, and for a new doctor, the training of the color ultrasonic examination needs to consume time and experience, and the training period is long; 3. the color ultrasonic examination is to make a prescribed action for a long time by a machine, and a doctor consumes physical strength in the examination process.
The existing color ultrasonic mechanical arm technology mainly comprises color ultrasonic inspection with a host machine and a probe separated, and color ultrasonic inspection with a probe clamped by the mechanical arm. At present, no contact-free color ultrasonic inspection is performed on the mechanical arm guided by visual and force feedback in the direction of the color ultrasonic inspection of the mechanical arm.
As in chinese patent 2011204856893, a medical ultrasound auxiliary device is disclosed, in which an ultrasound probe is mounted on the top of a mechanical arm, and an ultrasound inspection is performed by using a remote control manner, so as to reduce the repetitive labor of operators; chinese patent 2020103361331 discloses a virtual ultrasonic probe tracking method for remote control; the invention provides a remote-control virtual ultrasonic probe motion tracking method, which is characterized in that the pose of a virtual ultrasonic probe is captured by a depth camera, and depth data and pose data are corresponding to a motion mode by a deep learning method, so that remote automatic scanning of the ultrasonic probe is realized. The existing color Doppler ultrasound working mode is incomplete, and the condition of strong infectious diseases is not aimed at, and the existing technology is still carried out in the same inspection room, so that the non-contact color Doppler ultrasound inspection cannot be truly realized. In addition, the control mode of the mechanical arm is only single remote control, and the assistance of a visual algorithm is lacking; or the simple algorithm control, the accuracy and the stability are the problems to be solved. Meanwhile, the color Doppler ultrasound examination needs to find the specific angle and force of the probe on the patient to obtain a clear image, and the color Doppler ultrasound examination has no force feedback device to feed back the stress condition of the patient in real time, so that the safety is not guaranteed.
Disclosure of Invention
The invention provides a heart color ultrasound examination device and a method guided by vision and force feedback, which aims to overcome the defects in the prior art, ensure the accuracy of a doctor in remote control and improve the safety of color ultrasound examination.
In order to solve the technical problems, the invention adopts the following technical scheme: a heart color ultrasound examination device guided by vision and force feedback, comprising:
the mechanical arm is used for loading the color ultrasonic probe; receiving an instruction of a first industrial personal computer and completing a corresponding movement function; meanwhile, the acting force between the first industrial personal computer and the patient is fed back to the first industrial personal computer;
the first industrial personal computer is used for being connected with the mechanical arm and communicating with each other, and exchanging data with the second industrial personal computer; receiving force feedback information from the mechanical arm and transmitting the force feedback information to a second industrial personal computer;
the force feedback remote controller is used for remotely controlling the movement of the mechanical arm and sensing the force feedback from the mechanical arm;
the second industrial personal computer is used for being connected and communicated with the force feedback remote controller, communicating with the first industrial personal computer, receiving force feedback information transmitted by the first industrial personal computer, and transmitting the force feedback information to the force feedback remote controller so that the force feedback remote controller can feel force feedback from the mechanical arm; simultaneously, an instruction from the force feedback remote controller is sent to the first industrial personal computer to be transmitted to the mechanical arm;
the monitoring camera is connected and communicated with the first industrial personal computer and is used for monitoring the movement condition of the mechanical arm in real time;
the monocular camera is connected and communicated with the first industrial personal computer and used for collecting images of the end visual angles of the mechanical arm and returning the images to the first industrial personal computer, so that depth estimation can be carried out on the first industrial personal computer.
According to the invention, through the communication between the first industrial personal computer and the second industrial personal computer, the mechanical arm and the force feedback remote controller can be arranged in two independent rooms, so that remote color Doppler ultrasound is realized, and contact between a doctor and a patient is avoided; when in use; the doctor controls the force feedback remote controller, and the control information is transmitted to the mechanical arm through the second industrial personal computer and the second industrial personal computer so as to control the movement of the mechanical arm; simultaneously, the mechanical arm transmits the information of feeling the external force to the force feedback remote controller through the first industrial personal computer and the second industrial personal computer, a doctor can feel the effect of the force in real time on the force feedback remote controller, and simultaneously, the monitoring video can be checked in real time through the display screen of the second industrial personal computer so as to be convenient for carrying out the next operation.
Furthermore, a plurality of force sensors are arranged on the mechanical arm at intervals. And the external force information received by the mechanical arm is acquired in real time through the force sensor.
The invention also provides a heart color Doppler ultrasound examination method guided by vision and force feedback, which comprises the following steps:
the second industrial personal computer establishes a connection with the force feedback remote controller through a read-write program, reads control operation on the force feedback remote controller, and simultaneously transmits force feedback information to the force feedback remote controller, so that a manipulator can feel feedback force;
the second industrial personal computer establishes TCP communication with the first industrial personal computer, and video stream, remote control instructions and force feedback information are exchanged between the two industrial personal computers; the monocular camera transmits the shot image of the tail end of the mechanical arm to the first industrial personal computer, and the monitoring camera transmits the shot real-time condition of the movement of the mechanical arm to the first industrial personal computer; the first industrial personal computer transmits the video stream and the force feedback information to the second industrial personal computer, and the second industrial personal computer opens a video window and transmits a remote control instruction from the force feedback remote controller to the first industrial personal computer;
after the first industrial personal computer receives a remote control instruction from the second industrial personal computer, the instruction is issued to the ROS topic space, the rokae node controls the mechanical arm to correspondingly move in a remote control movement mode after subscribing the information of the remote control instruction, and meanwhile, the information of the force sensed by the mechanical arm is issued to the ROS space so as to be transmitted to the second industrial personal computer.
In the invention, a doctor operates a force feedback remote controller, a control instruction is transmitted to a mechanical arm through a second industrial personal computer and a first industrial personal computer, and the mechanical arm is loaded with a color ultrasonic probe to realize color ultrasonic inspection of a patient; on the other hand, the mechanical arm collects external force information in real time, and transmits force feedback information to the force feedback remote controller through the first industrial personal computer and the second industrial personal computer, so that a doctor can feel the force through the force feedback remote controller, and the doctor can feel the force so as to conveniently perform the next operation; meanwhile, the image information collected by the monocular camera and the monitoring camera is transmitted back to the second industrial personal computer through the first industrial personal computer, and a doctor can watch the shot video in real time through a display screen of the second industrial personal computer so as to clearly know the movement condition of the mechanical arm.
Further, the control of the mechanical arm specifically includes the following steps:
establishing communication between a first industrial personal computer and a mechanical arm: establishing TCP and UDP communication between the first industrial personal computer and the mechanical arm through an RCI external interface of the mechanical arm;
receive instructions under ROS space: a Rokae node for controlling the mechanical arm is established under the space of the first industrial personal computer and the ROS, the node subscribes to topics, and the topics contain control instructions from other nodes; the Rokae node receives the subscribed information and then enters a callback function, and a function built in the mechanical arm is called in the callback function so as to achieve the purpose of controlling the mechanical arm;
the mechanical arm returns information: the mechanical arm transmits the current coordinates and the pose and the external force sensed by each sensor on the mechanical arm to the first industrial personal computer after completing certain movement, and the first industrial personal computer receives and processes the information so as to transmit the information to the second industrial personal computer.
Further, when a function of starting motion is transmitted to the mechanical arm through the TCP, the position control of the Cartesian coordinate system is selected as a parameter, and the mechanical arm enters a position control mode, wherein the mechanical arm mainly depends on impedance to control the motion of the position under the Cartesian coordinate system; when the impedance coefficient is not 0, the mechanical arm can be dragged, but the mechanical arm can return to the original position by itself; when the impedance coefficient is set to 0, the mechanical arm can be freely dragged to any position without returning to the original position, so that the mechanical arm can be freely dragged.
Further, when the mechanical arm is controlled, the automatic movement of the mechanical arm is realized through the following steps:
the rokae node enables the mechanical arm to enter a single-position motion mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction under the ROS space, and reads and publishes the tail end coordinates of the mechanical arm in real time;
the motion planning node receives the information of the tail end coordinates of the mechanical arm, calculates and obtains the direction and the speed of the tail end of the mechanical arm which should move at the next moment according to the algorithm and the depth information issued by the depth estimation node, and issues the tail end coordinates of the mechanical arm to a control instruction subscribed by the rokae node;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
Further, when the mechanical arm is controlled, the remote control movement of the mechanical arm is realized through the following steps:
the remote control movement is under the control of a force feedback remote controller or the control of a keyboard;
the rokae node enables the mechanical arm to enter a single-position motion mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction in the ROS space, reads and issues the tail end coordinates of the mechanical arm and the external force applied to the mechanical arm in real time;
a keyboard or a force feedback remote controller reads a remote control operation instruction in real time or a remote control node reads the remote control operation instruction, and the instruction is issued to a control instruction topic;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
Furthermore, before performing color ultrasonic examination, the patient is far away from the color ultrasonic probe at the tail end of the mechanical arm, and the distance is likely to exceed the stroke of the force feedback remote control, and the distance automatically reaches a reasonable position before the doctor performs actual control on the mechanical arm through movement based on visual guidance, so that the doctor can conveniently perform remote control.
Further, the step of controlling the mechanical arm based on the visual guidance specifically includes:
the monocular camera mounted on the mechanical arm can simulate binocular vision by means of left and right slight movements of the tail end of the mechanical arm, and when the mechanical arm reciprocates between left and right points, the monocular camera can automatically shoot multiple groups of images to obtain binocular images under different base line lengths; the binocular image is transmitted to a computing node to perform parallax computation;
the first industrial personal computer receives Cartesian space coordinates of a left point and a right point returned by the mechanical arm, performs binocular calibration and matching at a computing node, and then performs parallax computation;
after obtaining the disparity map, according to the formula: depth = focal length x baseline/parallax, calculated to get a depth image; the depth images are obtained under different baselines to fuse the depth images, so that a defect-free and accurate depth image is obtained;
and detecting the position of a target point required by color Doppler ultrasound examination by using a target detection algorithm, judging the distance between the tail end of the mechanical arm and the target point of the patient according to the depth image, and deciding whether to approach the target point or to be far away from the target point.
Further, the algorithm used by the compute node is the sgbm algorithm in the opencv library.
Compared with the prior art, the beneficial effects are that: according to the heart color Doppler ultrasound examination device and method based on vision and force feedback guidance, the mechanical arm is remotely operated, and meanwhile, force feedback remote control is adopted, so that no risk of disease transmission exists between doctors and patients; and the force feedback remote control can convert the external force information received by the mechanical arm into the force feedback information received by the hands of the doctor in real time, thereby ensuring the accuracy of the doctor in remote control and improving the safety of color Doppler ultrasound examination. The invention also uses the visual guidance method to simplify the difficulty of doctor remote control, combines the visual guidance and the remote control, and greatly lightens the burden of repeated color ultrasound examination of doctors.
Drawings
Fig. 1 is a schematic diagram of the relationship between the various structures of the present invention.
FIG. 2 is a schematic diagram of the relationship of the motion of the control arm of the present invention.
Fig. 3 is a schematic structural diagram of the device for performing color Doppler ultrasound inspection.
Reference numerals: 1. a mechanical arm; 2. the first industrial personal computer; 3. a force feedback remote control; 4. the second industrial personal computer; 5. a monocular camera; 6. monitoring a camera; 7. a display screen; 8. color ultrasonic probe.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the invention; for the purpose of better illustrating the embodiments, certain elements of the drawings may be omitted, enlarged or reduced and do not represent the actual product dimensions; it will be appreciated by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted. The positional relationship described in the drawings are for illustrative purposes only and are not to be construed as limiting the invention.
As shown in fig. 1 and 3, a cardiac color ultrasound examination device guided by visual and force feedback includes:
the mechanical arm 1 is used for loading the color ultrasonic probe 8; receiving an instruction of a first industrial personal computer and completing a corresponding movement function; meanwhile, the acting force between the first industrial personal computer 2 and the patient is fed back;
the first industrial personal computer 2 is used for being connected with the mechanical arm 1 and communicating with each other, and exchanging data with the second industrial personal computer; receiving force feedback information from the mechanical arm 1 and transmitting the force feedback information to the second industrial personal computer 4;
a force feedback remote controller 3 for remotely controlling the motion of the mechanical arm 1 and sensing force feedback from the mechanical arm 1;
the second industrial personal computer 4 is used for being connected and communicated with the force feedback remote controller 3, communicating with the first industrial personal computer, receiving force feedback information transmitted by the first industrial personal computer 2, and transmitting the force feedback information to the force feedback remote controller 3 so that the force feedback remote controller 3 can feel force feedback from the mechanical arm 1; simultaneously, an instruction from the force feedback remote controller 3 is sent to the first industrial personal computer to be transmitted to the mechanical arm 1;
the monitoring camera 6 is connected and communicated with the first industrial personal computer and is used for monitoring the movement condition of the mechanical arm 1 in real time;
the monocular camera 5 is connected and communicated with the first industrial personal computer and is used for acquiring an image of the end visual angle of the mechanical arm 1 and returning the image to the first industrial personal computer 2, so that depth estimation can be carried out on the first industrial personal computer.
In the invention, the mechanical arm 1 and the force feedback remote controller 3 can be arranged in two independent rooms through the communication between the first industrial personal computer 2 and the second industrial personal computer 4, so that remote color Doppler ultrasound is realized, and contact between doctors and patients is avoided; when in use; the doctor controls the force feedback remote controller 3, and the control information is transmitted to the mechanical arm 1 through the second industrial personal computer 4 and the second industrial personal computer 4, so that the movement of the mechanical arm 1 is controlled; simultaneously, the mechanical arm 1 transmits the information of feeling the external force to the force feedback remote controller 3 through the first industrial personal computer 2 and the second industrial personal computer 4, and a doctor can observe the action of the real-time feeling force on the force feedback remote controller 3, and simultaneously, can view the monitoring video in real time through the display screen 7 of the second industrial personal computer 4 so as to perform the next operation.
Wherein, a plurality of force sensors are arranged on the mechanical arm 1 at intervals. External force information received by the mechanical arm 1 is acquired in real time through the force sensor.
In another embodiment, a method for cardiac color ultrasound examination by visual and force feedback guidance is provided, comprising the steps of:
the second industrial personal computer 4 establishes a connection with the force feedback remote controller 3 through a read-write program, the second industrial personal computer 4 reads the control operation on the force feedback remote controller 3 and simultaneously transmits the force feedback information to the force feedback remote controller 3, so that a controller can feel feedback force;
the second industrial personal computer 4 establishes TCP communication with the first industrial personal computer 2, and video stream, remote control instructions and force feedback information are exchanged between the two industrial personal computers; the monocular camera 5 transmits the photographed image of the tail end of the mechanical arm 1 to the first industrial personal computer 2, and the monitoring camera 6 transmits the photographed real-time condition of the movement of the mechanical arm 1 to the first industrial personal computer 2; the first industrial personal computer 2 transmits the video stream and the force feedback information to the second industrial personal computer 4, and the second industrial personal computer 4 opens a video window and transmits a remote control instruction from the force feedback remote controller 3 to the first industrial personal computer 2;
after the first industrial personal computer 2 receives the remote control instruction from the second industrial personal computer 4, the instruction is issued to the ROS topic space, the rokae node controls the mechanical arm 1 to correspondingly move in a remote control movement mode after subscribing the information of the remote control instruction, and meanwhile, the information of the force sensed by the mechanical arm 1 is issued to the ROS space so as to be transmitted to the second industrial personal computer 4.
In the invention, a doctor operates a force feedback remote controller 3, a control instruction is transmitted to a mechanical arm 1 through a second industrial personal computer 4 and a first industrial personal computer 2, and the mechanical arm 1 is loaded with a color ultrasonic probe 8 to realize color ultrasonic inspection of a patient; on the other hand, the mechanical arm 1 collects external force information in real time, and transmits force feedback information to the force feedback remote controller 3 through the first industrial personal computer 2 and the second industrial personal computer 4, so that a doctor can feel the force through the force feedback remote controller 3, and thus the doctor can feel the force so as to be convenient for carrying out the next operation; meanwhile, the image information collected by the monocular camera 5 and the monitoring camera 6 is transmitted back to the second industrial personal computer 4 through the first industrial personal computer 2, and a doctor can watch a shot video in real time through the display screen 7 of the second industrial personal computer 4 so as to clearly know the movement condition of the mechanical arm 1.
In one embodiment, the control of the mechanical arm 1 specifically includes the following steps:
establishing communication between the first industrial personal computer 2 and the mechanical arm 1: establishing TCP and UDP communication between the first industrial personal computer and the mechanical arm 1 through an RCI external interface of the mechanical arm 1;
receive instructions under ROS space: a Rokae node for controlling the mechanical arm 1 is established under the space of the first industrial personal computer 2 and the ROS, the node subscribes to topics, and the topics contain control instructions from other nodes; the Rokae node receives the subscribed information and then enters a callback function, and a function built in the mechanical arm 1 is called in the callback function so as to achieve the purpose of controlling the mechanical arm 1;
the mechanical arm 1 returns information: after completing a certain movement, the mechanical arm 1 transmits the current coordinates and the pose and the external force sensed by each sensor on the mechanical arm 1 to the first industrial personal computer 2, and the first industrial personal computer 2 receives and processes the information to transmit the information to the second industrial personal computer 4.
In one embodiment, the cartesian coordinate system position control is selected as a parameter when the function of starting the motion is transmitted to the mechanical arm 1 through the TCP, and the mechanical arm 1 enters a position control mode, wherein the mechanical arm 1 mainly performs motion control on the position under the cartesian coordinate system depending on the impedance; when the impedance coefficient is not 0, the mechanical arm 1 can be dragged, but the mechanical arm 1 can return to the original position by itself; when the impedance coefficient is set to 0, the mechanical arm 1 can be freely dragged to any position without returning to the original position, so that the mechanical arm 1 can be freely dragged.
In one embodiment, when the mechanical arm 1 is controlled, the automatic movement of the mechanical arm 1 is achieved by:
the rokae node enables the mechanical arm 1 to enter a single movement mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction under the ROS space, and reads and publishes the tail end coordinate of the mechanical arm 1 in real time;
the motion planning node receives the information of the tail end coordinates of the mechanical arm 1, calculates the direction and the speed of the tail end of the mechanical arm 1, which should move at the next moment, according to the algorithm and the depth information issued by the depth estimation node, and issues the tail end coordinates of the mechanical arm 1 to a control instruction subscribed by the rokae node;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
In one embodiment, when the mechanical arm 1 is controlled, the remote control movement of the mechanical arm 1 is realized by the following steps:
the remote control movement is under the control of the force feedback remote controller 3 or the keyboard;
the rokae node enables the mechanical arm 1 to enter a single movement mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction under the ROS space, reads and issues the tail end coordinate of the mechanical arm 1 and the external force born by the mechanical arm 1 in real time;
the keyboard or force feedback remote controller 3 reads the remote control operation instruction in real time by the node or the remote control node, and issues the instruction to the topic of control instruction;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
In one embodiment, before performing the color Doppler ultrasound examination, the patient is far away from the color Doppler ultrasound probe 8 at the tail end of the mechanical arm 1, and the distance is likely to exceed the travel of the force feedback remote control, and the distance is automatically reached to a reasonable position before the doctor actually controls the mechanical arm 1 through the movement based on visual guidance, so that the doctor can conveniently perform remote control.
In one embodiment, the step of controlling the mechanical arm 1 based on the visual guidance specifically includes:
the monocular camera 5 mounted on the mechanical arm 1 simulates binocular vision by means of left and right slight movement of the tail end of the mechanical arm 1, and when the mechanical arm 1 reciprocates between left and right points, the monocular camera 5 can automatically shoot multiple groups of images to obtain binocular images under different base line lengths; the binocular image is transmitted to a computing node to perform parallax computation;
the first industrial personal computer receives Cartesian space coordinates of the left point and the right point returned by the mechanical arm 1, performs binocular calibration and matching at a computing node, and then performs parallax computation; the algorithm used by the compute node is the sgbm algorithm in the opencv library.
After obtaining the disparity map, according to the formula: depth = focal length x baseline/parallax, calculated to get a depth image; the depth images are obtained under different baselines to fuse the depth images, so that a defect-free and accurate depth image is obtained;
and detecting the position of a target point required by color Doppler ultrasound examination by using a target detection algorithm, judging the distance between the tail end of the mechanical arm 1 and the target point of the patient according to the depth image, and deciding whether to approach the target point or to be far away from the target point.
While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.
It is to be understood that the above examples of the present invention are provided by way of illustration only and not by way of limitation of the embodiments of the present invention. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are desired to be protected by the following claims.
Claims (8)
1. A heart color ultrasound examination device guided by vision and force feedback, comprising:
the mechanical arm (1) is used for loading the color ultrasonic probe (8); receiving an instruction of a first industrial personal computer and completing a corresponding movement function; meanwhile, the acting force between the first industrial personal computer and the patient is fed back to the first industrial personal computer (2);
the first industrial personal computer (2) is used for being connected with the mechanical arm (1) and communicating with each other, and exchanging data with the second industrial personal computer; receiving force feedback information from the mechanical arm (1) and transmitting the force feedback information to the second industrial personal computer (4);
a force feedback remote controller (3) for receiving a control operation of a manipulator to remotely control the movement of the mechanical arm (1) and sensing a force feedback from the mechanical arm (1) after the mechanical arm (1) automatically reaches a reasonable position by the movement based on visual guidance;
the second industrial personal computer (4) is used for being connected and communicated with the force feedback remote controller (3) and communicating with the first industrial personal computer, receiving force feedback information transmitted by the first industrial personal computer (2) and transmitting the force feedback information to the force feedback remote controller (3) so that the force feedback remote controller (3) can feel force feedback from the mechanical arm (1); simultaneously, an instruction from the force feedback remote controller (3) is sent to the first industrial personal computer to be transmitted to the mechanical arm (1);
the monitoring camera (6) is connected and communicated with the first industrial personal computer and is used for monitoring the movement condition of the mechanical arm (1) in real time;
the monocular camera (5) is connected and communicated with the first industrial personal computer and is used for collecting images of the end view angle of the mechanical arm (1) and returning the images to the first industrial personal computer (2) so as to perform depth estimation on the first industrial personal computer, wherein before performing color ultrasonic inspection, if the distance between a patient and a color ultrasonic probe (8) at the end of the mechanical arm (1) exceeds the stroke of force feedback remote control, the monocular camera (5) moves slightly left and right by virtue of the end of the mechanical arm (1) to simulate binocular vision, and when the mechanical arm (1) reciprocates between left and right points, the monocular camera (5) automatically shoots a plurality of groups of images so as to obtain binocular images under different base line lengths; the binocular image is transmitted to a computing node to perform parallax computation;
the first industrial personal computer receives Cartesian space coordinates of a left point and a right point returned by the mechanical arm (1), performs binocular calibration and matching at a computing node, and then performs parallax calculation; after obtaining the disparity map, according to the formula: depth = focal length x baseline/parallax, calculated to get a depth image; the depth images obtained under different baselines are used for fusion of the depth images, so that a defect-free and accurate depth image is obtained; detecting the position of a target point required by color Doppler ultrasound examination by using a target detection algorithm, judging the distance between the tail end of the mechanical arm (1) and the target point of a patient according to a depth image, and deciding whether the mechanical arm (1) is close to the target point or far from the target point so that the mechanical arm (1) automatically reaches a reasonable position before a manipulator actually manipulates the mechanical arm (1);
the second industrial personal computer (4) further comprises a display screen (7) used for watching videos shot by the monocular camera (5) and the monitoring camera (6) in real time.
2. The heart color Doppler ultrasound examination device guided by vision and force feedback according to claim 1, wherein a plurality of force sensors are arranged on the mechanical arm (1) at intervals.
3. The heart color Doppler ultrasound examination method guided by vision and force feedback is characterized by comprising the following steps of:
before performing color ultrasonic inspection, if the distance between a patient and a color ultrasonic probe (8) at the tail end of the mechanical arm (1) exceeds a force feedback remote control stroke, the mechanical arm (1) is guided to automatically reach a reasonable position before a manipulator actually manipulates the mechanical arm (1) by a motion based on visual guidance, and the method comprises the following steps:
the monocular camera (5) mounted on the mechanical arm (1) moves slightly left and right by means of the tail end of the mechanical arm (1) to simulate binocular vision, and when the mechanical arm (1) reciprocates between the left point and the right point, the monocular camera (5) automatically shoots multiple groups of images to obtain binocular images under different base line lengths; the binocular image is transmitted to a computing node to perform parallax computation;
the first industrial personal computer receives Cartesian space coordinates of left and right points returned by the mechanical arm (1), performs binocular calibration and matching at a computing node, and then performs parallax calculation; after obtaining the disparity map, according to the formula:
depth = focal length x baseline/parallax, calculated to get a depth image;
fusion of depth images is performed by using the depth images obtained under different baselines to obtain a defect-free image,
a more accurate depth image;
detecting the position of a target point required by color Doppler ultrasound examination by using a target detection algorithm, judging the distance between the tail end of the mechanical arm (1) and the target point of a patient according to a depth image, and deciding whether the mechanical arm (1) is close to the target point or far from the target point so that the mechanical arm (1) automatically reaches a reasonable position before a manipulator actually manipulates the mechanical arm (1);
the force feedback remote controller (3) receives the control operation of a manipulator after the mechanical arm (1) automatically reaches a reasonable position through movement based on visual guidance;
the second industrial personal computer (4) establishes a connection with the force feedback remote controller (3) through a read-write program, and the second industrial personal computer (4) reads control operation on the force feedback remote controller (3) and simultaneously transmits force feedback information to the force feedback remote controller (3) so that a controller can feel feedback force;
the second industrial personal computer (4) establishes TCP communication with the first industrial personal computer (2), and video stream, remote control instructions and force feedback information are exchanged between the two industrial personal computers; the monocular camera (5) transmits the photographed image of the tail end of the mechanical arm (1) to the first industrial personal computer (2), and the monitoring camera (6) transmits the photographed real-time condition of the movement of the mechanical arm (1) to the first industrial personal computer (2); the first industrial personal computer (2) transmits the video stream and the force feedback information to the second industrial personal computer (4), and the second industrial personal computer (4) opens a video window and transmits a remote control instruction from the force feedback remote controller (3) to the first industrial personal computer (2); the second industrial personal computer (4) further comprises a display screen (7) for watching videos shot by the monocular camera (5) and the monitoring camera (6) in real time;
after the first industrial personal computer (2) receives a remote control instruction from the second industrial personal computer (4), the instruction is issued to the ROS topic space, the rokae node controls the mechanical arm (1) to correspondingly move in a remote control movement mode after subscribing the information of the remote control instruction, and meanwhile, the information of the force sensed by the mechanical arm (1) is issued to the ROS space so as to be transmitted to the second industrial personal computer (4).
4. A method of cardiac color ultrasound examination by means of visual and force feedback guidance according to claim 3, characterized in that the control of the robotic arm (1) specifically comprises the steps of:
establishing communication between a first industrial personal computer (2) and a mechanical arm (1): establishing TCP and UDP communication between the first industrial personal computer and the mechanical arm (1) through an RCI external interface of the mechanical arm (1);
receive instructions under ROS space: a Rokae node for controlling the mechanical arm (1) is established under the space of the first industrial personal computer (2) and the ROS, the node subscribes to topics, and the topics contain control instructions from other nodes; the Rokae node receives the subscribed information and then enters a callback function, and a function built in the mechanical arm (1) is called in the callback function so as to achieve the purpose of controlling the mechanical arm (1);
the mechanical arm (1) returns information: the mechanical arm (1) transmits the current coordinates and the pose and the external force sensed by each sensor on the mechanical arm (1) to the first industrial personal computer (2) after completing certain movements, and the first industrial personal computer (2) receives and processes information so as to transmit the information to the second industrial personal computer (4).
5. The method for cardiac color Doppler ultrasound examination by means of visual and force feedback guidance according to claim 4, wherein,
selecting a Cartesian coordinate system position control as a parameter when a function of starting motion is transmitted to the mechanical arm (1) through a TCP, and enabling the mechanical arm (1) to enter a position control mode, wherein the mechanical arm (1) mainly relies on impedance to perform motion control on the position under the Cartesian coordinate system; when the impedance coefficient is not 0, the mechanical arm (1) can be dragged, but the mechanical arm (1) can return to the original position by itself; when the impedance coefficient is set to 0, the mechanical arm (1) can be freely dragged to any position without returning to the original position, so that the mechanical arm (1) can be freely dragged.
6. The method for cardiac color ultrasound examination by means of visual and force feedback guidance according to claim 4, wherein the automatic movement of the mechanical arm (1) is achieved by the following steps when the mechanical arm (1) is controlled:
the rokae node enables the mechanical arm (1) to enter a single movement mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction under the ROS space, and reads and publishes the tail end coordinate of the mechanical arm (1) in real time;
the motion planning node receives a message of the tail end coordinates of the mechanical arm (1), calculates the direction and the speed of the tail end of the mechanical arm (1) which should move at the next moment according to an algorithm and depth information issued by the depth estimation node, and issues the tail end coordinates of the mechanical arm (1) to a control instruction subscribed by a rokae node;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
7. The heart color Doppler ultrasound examination method by means of vision and force feedback guidance according to claim 4, wherein when the mechanical arm (1) is controlled, the remote control movement of the mechanical arm (1) is realized by the following steps:
the remote control movement is under the control of a force feedback remote controller (3) or the control of a keyboard;
the rokae node enables the mechanical arm (1) to enter a single movement mode under a Cartesian coordinate system through a built-in function; meanwhile, on another thread, the rokae node starts to subscribe to the topic of a control instruction under the ROS space, reads and issues the tail end coordinates of the mechanical arm (1) and the external force born by the mechanical arm (1) in real time;
a keyboard or a force feedback remote controller (3) reads a node or a remote control node and reads a remote control operation instruction in real time, and the instruction is issued to a control instruction topic;
the rokae enters a callback function after receiving the message, and moves towards a determined direction according to the content of the message.
8. A method of conducting cardiac ultrasound examinations by means of visual and force feedback according to claim 3 wherein the algorithm used by the compute nodes is the sgbm algorithm in opencv library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110776269.9A CN113499094B (en) | 2021-07-08 | 2021-07-08 | Heart color Doppler ultrasound examination device and method guided by vision and force feedback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110776269.9A CN113499094B (en) | 2021-07-08 | 2021-07-08 | Heart color Doppler ultrasound examination device and method guided by vision and force feedback |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113499094A CN113499094A (en) | 2021-10-15 |
| CN113499094B true CN113499094B (en) | 2023-07-25 |
Family
ID=78012410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110776269.9A Active CN113499094B (en) | 2021-07-08 | 2021-07-08 | Heart color Doppler ultrasound examination device and method guided by vision and force feedback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113499094B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114469340B (en) * | 2022-01-18 | 2024-05-28 | 上海健康医学院 | Surgical robot soft tissue recognition processing method and system |
| CN114789445B (en) * | 2022-05-25 | 2023-07-07 | 广东轻工职业技术学院 | Manipulator system and method applied to remote heart color Doppler ultrasound robot |
| CN116077089B (en) * | 2023-02-28 | 2023-07-07 | 北京智源人工智能研究院 | Multimode safety interaction method and device for ultrasonic scanning robot |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107693050A (en) * | 2017-11-23 | 2018-02-16 | 复旦大学附属中山医院 | A kind of telemedicine ultrasonic examination system and method |
| CN107767423A (en) * | 2017-10-10 | 2018-03-06 | 大连理工大学 | A kind of mechanical arm target positioning grasping means based on binocular vision |
| CN109048926A (en) * | 2018-10-24 | 2018-12-21 | 河北工业大学 | A kind of intelligent robot obstacle avoidance system and method based on stereoscopic vision |
| CN109288540A (en) * | 2017-07-24 | 2019-02-01 | 云南师范大学 | A kind of long-distance ultrasonic diagnosis system with touch feedback |
| CN109357630A (en) * | 2018-10-30 | 2019-02-19 | 南京工业大学 | A kind of polymorphic type batch workpiece vision measurement system and method |
| CN112132894A (en) * | 2020-09-08 | 2020-12-25 | 大连理工大学 | Mechanical arm real-time tracking method based on binocular vision guidance |
| CN112998749A (en) * | 2021-02-18 | 2021-06-22 | 江苏省人民医院（南京医科大学第一附属医院） | Automatic ultrasonic inspection system based on visual servoing |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2104455A2 (en) * | 2006-09-25 | 2009-09-30 | Koninklijke Philips Electronics N.V. | Haptic feedback medical scanning methods and systems |
| DE102007046700A1 (en) * | 2007-09-28 | 2009-04-16 | Siemens Ag | ultrasound device |
| TWI454246B (en) * | 2011-09-30 | 2014-10-01 | Mackay Memorial Hospital | Immediate monitoring of the target location of the radiotherapy system |
| WO2015135057A1 (en) * | 2014-03-14 | 2015-09-17 | Synaptive Medical (Barbados) Inc. | Intelligent positioning system and methods therefore |
| CN104820418B (en) * | 2015-04-22 | 2018-04-24 | 遨博（北京）智能科技有限公司 | A kind of embedded vision system and its application method for mechanical arm |
| US10609354B2 (en) * | 2016-02-12 | 2020-03-31 | Sony Corporation | Medical image processing device, system, method, and program |
| US20170252002A1 (en) * | 2016-03-07 | 2017-09-07 | Toshiba Medical Systems Corporation | Ultrasonic diagnostic apparatus and ultrasonic diagnosis support apparatus |
| TWI567693B (en) * | 2016-05-17 | 2017-01-21 | 緯創資通股份有限公司 | Method and system for generating depth information |
| US11647983B2 (en) * | 2017-05-05 | 2023-05-16 | International Business Machines Corporation | Automating ultrasound examination of a vascular system |
| CN107526681B (en) * | 2017-08-11 | 2020-05-26 | 上海木木聚枞机器人科技有限公司 | Robot test system and test method |
| CN107883929B (en) * | 2017-09-22 | 2020-01-31 | 中冶赛迪技术研究中心有限公司 | Monocular vision positioning device and method based on multi-joint mechanical arm |
| US20200194117A1 (en) * | 2018-12-13 | 2020-06-18 | University Of Maryland, College Park | Systems, methods, and media for remote trauma assessment |
| CN110216698A (en) * | 2019-03-11 | 2019-09-10 | 浙江工业大学 | A kind of mechanical arm remote control system based on ROS |
| CN111084638A (en) * | 2020-02-21 | 2020-05-01 | 常州市第二人民医院 | Ultrasonic probe surface pressure detection device |
| CN111544039A (en) * | 2020-05-19 | 2020-08-18 | 南京信息职业技术学院 | Working method of color Doppler ultrasound examination cooperation device |
| CN112706164B (en) * | 2020-12-18 | 2022-05-24 | 深圳市大富智慧健康科技有限公司 | Automatic correction method, device and equipment for initial pose of mechanical arm and storage medium |
| CN112650120A (en) * | 2020-12-22 | 2021-04-13 | 华中科技大学同济医学院附属协和医院 | Robot remote control system, method and storage medium |
| CN112936278B (en) * | 2021-02-07 | 2022-07-29 | 深圳市优必选科技股份有限公司 | Man-machine cooperation control method and device for robot and robot |
-
2021
- 2021-07-08 CN CN202110776269.9A patent/CN113499094B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109288540A (en) * | 2017-07-24 | 2019-02-01 | 云南师范大学 | A kind of long-distance ultrasonic diagnosis system with touch feedback |
| CN107767423A (en) * | 2017-10-10 | 2018-03-06 | 大连理工大学 | A kind of mechanical arm target positioning grasping means based on binocular vision |
| CN107693050A (en) * | 2017-11-23 | 2018-02-16 | 复旦大学附属中山医院 | A kind of telemedicine ultrasonic examination system and method |
| CN109048926A (en) * | 2018-10-24 | 2018-12-21 | 河北工业大学 | A kind of intelligent robot obstacle avoidance system and method based on stereoscopic vision |
| CN109357630A (en) * | 2018-10-30 | 2019-02-19 | 南京工业大学 | A kind of polymorphic type batch workpiece vision measurement system and method |
| CN112132894A (en) * | 2020-09-08 | 2020-12-25 | 大连理工大学 | Mechanical arm real-time tracking method based on binocular vision guidance |
| CN112998749A (en) * | 2021-02-18 | 2021-06-22 | 江苏省人民医院（南京医科大学第一附属医院） | Automatic ultrasonic inspection system based on visual servoing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113499094A (en) | 2021-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113499094B (en) | Heart color Doppler ultrasound examination device and method guided by vision and force feedback | |
| US11612446B2 (en) | Systems, methods, and computer-readable program products for controlling a robotically delivered manipulator | |
| US11534246B2 (en) | User input device for use in robotic surgery | |
| WO2023056670A1 (en) | Mechanical arm autonomous mobile grabbing method under complex illumination conditions based on visual-tactile fusion | |
| WO2020221311A1 (en) | Wearable device-based mobile robot control system and control method | |
| KR102358967B1 (en) | Systems and methods for control of imaging instrument orientation | |
| EP4306057A1 (en) | Portable remote ultrasound scanning system and compliance control method for safe ultrasound scanning | |
| US20210132681A1 (en) | Natural human-computer interaction system based on multi-sensing data fusion | |
| CN112618026B (en) | Remote operation data fusion interactive display system and method | |
| US20230042756A1 (en) | Autonomous mobile grabbing method for mechanical arm based on visual-haptic fusion under complex illumination condition | |
| EP3941693A1 (en) | Robotic surgical collision detection systems | |
| KR20180068336A (en) | Surgical system with training or auxiliary functions | |
| CN111438673B (en) | High-altitude operation teleoperation method and system based on stereoscopic vision and gesture control | |
| KR20090113084A (en) | Method and system for motion control in humanoid robot | |
| WO2022002159A1 (en) | Master-slave motion control method, robot system, device, and storage medium | |
| CA2647435A1 (en) | Intelligent interface device for grasping of an object by a manipulating robot and method of implementing this device | |
| JP2011110620A (en) | Method of controlling action of robot, and robot system | |
| CN110682291B (en) | Robot teleoperation system based on VR and teleoperation method thereof | |
| Seo et al. | A comparative study of in-field motion capture approaches for body kinematics measurement in construction | |
| CN111216109A (en) | Visual following device and method for clinical treatment and detection | |
| WO2023237105A1 (en) | Method for displaying virtual surgical instrument on surgeon console, and surgeon console | |
| CN112069873A (en) | LeapMotion gesture recognition-based screen control system and method | |
| CN112650120A (en) | Robot remote control system, method and storage medium | |
| CN207202885U (en) | A kind of ultrasonic testing system | |
| US20200179070A1 (en) | Robotic surgical system control arm including dual encoders |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |