WO2023024396A1

WO2023024396A1 - Recognition, autonomous positioning and scanning method for visual image and medical image fusion

Info

Publication number: WO2023024396A1
Application number: PCT/CN2022/000117
Authority: WO
Inventors: 谈斯聪; 于皓; 于梦非
Original assignee: 谈斯聪
Priority date: 2021-08-23
Filing date: 2022-08-18
Publication date: 2023-03-02
Also published as: CN113855067A; AU2022335276A1; CN118338850A

Abstract

Recognition, autonomous positioning and scanning methods for visual image and medical image fusion, using artificial intelligence robot technology. Provided are a method for autonomous positioning and recognition of a human organ feature position to collect an ultrasonic image, remotely and adaptively collecting an image and a video of the neck, the left and right lobes and isthmus of the thyroid, and the parotid gland; a method for remotely and adaptively collecting and scanning lower limb blood vessels, collecting a medical image and video of organs and tissues; and a scanning means for remotely and adaptively capturing and scanning skeletal joints, muscles, and nerves. By means of autonomous and remote controlled collecting and sharing of medical images, image sharing is implemented, thereby alleviating problems of medical care operations such as high stress and increased night shifts. This improves remote collection and sharing of medical images, consultations ward rounding efficiency, and using expert opinion to address clinical cases. The invention is applicable to outpatient clinics, wards, and overseas medical institutions.

Description

Visual image and medical image fusion recognition, autonomous positioning scanning method

technical field

The invention belongs to the technical field of medical artificial intelligence, and relates to data analysis technology, the technical field of robot action planning, an image intelligent recognition method, artificial intelligence and medical data analysis and recognition technology and method.

Background technique

Currently used in the medical field, during the inspection process, due to the analysis of various human factors, the quality of medical image and video acquisition is poor, the degree of standardization is low, and the accuracy of identifying the disease is poor. The fields of specialists and medical specialties are limited, and the administrator controls remotely, using the robotic arm, vision device, depth vision device and various neural network methods and improvement methods carried by the robot to intelligently recognize faces, human organs, bones, Auxiliary collection of medical images and videos.

During the epidemic period, the infectivity is high, the collection infection risk is high, the efficiency is low, manual collection is not accurate, and manual collection will lead to problems such as the spread of the plague. Using the remote and autonomous collection of medical images and videos of the robotic arm to achieve remote collection and autonomous collection, Intelligent identification and analysis of data, images, videos, autonomous positioning, and scanning can effectively prevent the spread of infectious diseases, plague and other major diseases.

Intelligent recognition of visual image and medical image fusion, autonomous positioning and scanning methods, adaptive adjustment of robotic arms, intelligent recognition of human body, organs, bones, recognition, remote and autonomous mobile robotic arms, scanning probes, scanning devices, autonomous positioning , Scan parts, organs, according to the intelligent scanning method, autonomously scan, collect medical images, videos, greatly improve the efficiency of intelligent identification, collection of medical data, medical images and videos.

technical problem

The purpose of the present invention is to overcome the shortcomings and deficiencies of the above-mentioned prior art, to provide a device for collecting medical images by an autonomous robot, as well as the intelligent recognition of the fusion of visual images and medical images of the present invention, and the autonomous positioning and scanning of robotic arms The method solves the problem of man-made scanning, inspection, collection, diagnosis and treatment errors.

A kind of intelligent identification method for the fusion of multiple medical images to intelligently identify human facial features, organs, bones, joints, blood vessels, human body features and their positions, as well as robots, robotic arms, and medical devices for autonomous positioning, external position areas of the human body, and scanning to collect medical images The method has become the key technical issue of self-positioning and scanning.

Effective scanning and complete scanning methods have become another important technical problem in ultrasonic scanning. The present invention provides a method for remote and self-adaptive scanning of the neck, left and right thyroid lobes, isthmus, and parotid gland, and image and video collection; a remote and self-adaptive scanning method for collecting blood vessels of the lower extremities, collecting organs and tissues A medical image and video method; a remote and self-adaptive method for collecting bones, joints, muscles, and nerve scans, collecting organs, and organizing medical images and videos.

The technical scheme adopted in the present invention

Fusion of visual images and medical images to intelligently identify human facial features, organs, bones, joints, blood vessels, human body features and their positions. The method is to integrate general visual images, depth images, ultrasonic images, and endoscopic images A method for intelligently identifying human organs, bones, joints, blood vessels, human body feature points and their positions and a method for autonomously positioning human organs, bones, joints, blood vessels, and human body feature points, including the following steps:

S1. Establish general visual image models, human organ models, blood vessel models and feature models. The general image models include: face, facial features, ears, lips, mandible, neck, navel, nipples, characteristic genitalia, characteristic parts and Its position is used as a feature item, and the neural network algorithm and its improved method are used to intelligently identify the human body's features and its position.

S2. Establish a depth vision device, a depth vision image model, and a skeleton model.

S3. Combining the depth information, the position information of the human body where the bones are located, and the information of the adjacent characteristic organs identified by the general image model described in S1 as the feature items of the intelligent bone recognition model as input items.

S4. Apply the neural network algorithm and its improved method to intelligently identify each bone and the position of each bone, mandible, costal bottom bone, xiphoid process, spine, spine position, bone lower boundary position, shoulder joint, knee joint, foot, Bones, foot joints, lumbar joints and their positions, the position of each joint.

S5. Standardize the adjacent characteristic organ information identified by the general image model described in S1 as external scanning feature information, and externally scan the bone information for the bone information identified in S4,

S6. Establish a feature model under the ultrasound image, and combine the color information of blood vessels under the ultrasound image, the location information of blood vessels, and the contour shape features, structural features, and color features of organs in the ultrasound image.

S7. Combining blood vessel information, blood vessel position information with organ information and organ features under the ultrasound image into an information combination item as a feature item and an input item of the ultrasound image model.

S8. Using the external scan feature information and the external scan bone information as the external scan information, combine the adjacent characteristic organ information identified by the general image model described in S1 with the bone information identified by S4 and the features of the ultrasound image model described in S7 The item and its location area are input into the neural network and its improved method and weight optimizer, and the output value is obtained through image training.

S9. Improve the deep neural network method and the weight optimizer, and obtain the output value and the identification results of organs, blood vessels, bones, human body scan organs and position information through image training.

S10. Outputting the result, as the result of external scanning to autonomously locate organs, blood vessels, bones, and their positions in the human body.

A method for a robot to autonomously locate the coordinates of an external position area of a human body, scan and collect medical images, comprising the following steps:

S1. According to the administrator, doctor communication module, robot, and medical device to issue acquisition tasks and doctor's order messages, obtain the organs of the human body acquisition images corresponding to the acquisition tasks and their external position area coordinates, set them as the target, set the target name, target parameters, Location information, set the communication target.

S2. The robot vision acquisition device and visual recognition module release the external features corresponding to each organ, the external scanning feature information of the nearby characteristic organs recognized by the general image model The coordinates of the external position area of the human body; the depth information released by the depth camera, the external bone information of the nearby neighbors Scan bone information.

S3. The robot arm, ultrasound device, ultrasound probe, robot, and medical device subscribe to the location information of the corresponding organ external scanning area,

Subscribe target, parameter, target pose, pose marker, set target for head id, target pose, direction value, set timestamp

S4. The remote main control system and the ultrasonic probe mounted on the autonomous robot arm move and scan the human body acquisition area according to the subscribed acquisition area location and the action of the robotic arm image acquisition action planning module. The ultrasonic probe and the ultrasonic device release the collected image information, blood vessel color information, blood vessel position information and contour shape features, structural features, and color features of the organs in the ultrasonic image.

S5. The robot, the medical device and the visual recognition module subscribe to the image information. According to claim 8, the blood vessel color information, blood vessel position information, and ultrasonic image organ contour shape features, structural features, and color features are input into the calculation model according to claim 8, and intelligently identify whether the image is a target organ tissue according to claim 8. If it is a scan Check the target organ.

S6. Set the acquisition target parameters (pose mark, time stamp, target for head id, COG target pose, direction value), set the allowable error of position and attitude, when the motion planning fails, allow re-planning, set the target position The reference coordinate system, set the time limit for each motion planning.

S7. Robots, medical devices, and the remote end of the robot arm and self-adaptive adjustments to collect image parameters of medical devices, video parameters, image collection methods, whether they meet the image and video recognition standards, and whether the collection is effective.

S8. Robots, medical devices, remote ends of robotic arms, and self-adaptive adjustments to the scanning mode, probe angle, and parameters of the ultrasound probe, to check the collection position of the target organ, and whether all the images and videos of the collected organs and tissues are in the scanning mode. The image, video, whether it is the target organ, the complete collection of the tissue.

S9. Return target organ acquisition completion information, robot, medical device and robotic arm subscribe task information, robot, medical device and robotic arm distal end and adaptively move the ultrasonic probe to the external scanning position area of the next target organ.

S10. The robot and the medical device determine that all target organ collection tasks are completed according to the returned target organ collection completion information.

A method for remote and adaptive scanning of the neck, left and right thyroid lobes, isthmus, and parotid gland, and collection of medical images and videos:

S1. According to claim 6, according to the general visual device and a variety of medical image fusion intelligent recognition methods, intelligently identify the human ear, auricle and its position, intelligently identify the human lip and its position, through the depth vision acquisition device and multiple A medical image fusion intelligent identification method, intelligent identification of bones, intelligent identification of the mandible at the bottom of the ear, spine, bone position, spine position, intelligent identification of bones, and the position of bones below the midline of the lip.

S2. According to claim 7, according to the autonomous positioning of the robot, the coordinates of the external position area of the human body, the method of scanning and collecting medical images, the robot arm, the ultrasonic device, the ultrasonic probe, the robot, and the medical device subscribe to the neck, the left and right thyroid lobes, the isthmus, and the parotid gland Location information of the external scanning area, scanning method, probe angle, subscription target, parameters, target pose, pose mark, set target for head id, target pose, direction value, set timestamp.

S3. Robots, medical devices and robotic arms remote and self-adaptive adjustment to collect image parameters, gain parameters, color gain parameters, sensitivity time control adjustment parameters, time gain control parameters, focus parameters, depth parameters, frame size, Blood flow velocity scale parameters, video parameters, image acquisition method, robotic arm, ultrasound probe, remote control and adaptive adjustment of probe rotation angle, tilt angle, ultrasound probe scanning method, probe angle, parameters to the following organs, tissues Effective collection, complete collection.

S4. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans along the lower boundary of the bone at the bottom of the ear, and collects images and videos of the neck, spine, common carotid artery, internal carotid artery, and external carotid artery.

S5. Remote control and self-adaptive mobile robot arm, the ultrasonic probe moves along the bone position below the midline of the lip, along the carotid artery from the head side to the foot side to scan the left and right lobes and isthmus of the thyroid gland, and collect medical images and videos.

S6. Remote control and adaptive mobile robot arm, ultrasonic probe, along the bottom of the ear, bone position, along the mandible, bone position, scan the parotid gland, submandibular gland, sublingual gland to the bone position below the midline of the lip, collect medical treatment images and videos.

A method for remote and self-adaptive scanning of blood vessels of lower extremities, collecting medical images and videos of blood vessels of lower extremities, organs and tissues:

S1. According to claim 6, according to the general visual device and multiple medical image fusion intelligent recognition methods, intelligently identify human ears, auricles and their positions, and intelligently identify navel and their positions.

S2. Through the deep vision acquisition device and various medical image fusion intelligent recognition methods, intelligently identify bones, rib bottom bones and xiphoid process positions, ischium, knee joints and their positions, dorsally measure popliteal fossa and its positions, foot bones, Foot joints and their positions.

S3. According to claim 6, through the ultrasonic image and multiple medical image fusion intelligent recognition methods, the blood vessel color information, blood vessel position information and organ information under the ultrasonic image are intelligently identified for abdominal aorta, groin, superficial femoral artery, popliteal artery, Anterior tibial artery, posterior tibial artery.

S4. According to claim 6, the pulsation position is identified as the groin area by using the ultrasonic image and a variety of medical image fusion intelligent identification methods, and using the dynamic identification method under the ultrasonic image.

S5. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the groin area, the ultrasonic probe scans the groin area from the position of the xiphoid process down to the navel and along the abdominal aorta to collect medical images and videos.

S6. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans the superficial femoral artery along the groin area, moves to the knee joint and its position, measures the position of the popliteal fossa from the back, and moves from the position of the popliteal fossa to the dorsal side of the knee joint Vascular scanning, collecting medical images and videos.

S7. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans downward along the popliteal artery, anterior tibial artery, and posterior tibial artery, presses with a pressing device, scans nearby blood vessels along the anterior tibial artery from the knee joint, and collects medical records. images and videos.

S8. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the foot bones, foot joints and their positions, scan the dorsalis pedis artery along the dorsum of the foot, scan the plantar artery along the sole of the foot, and collect medical images and videos.

S9. Remotely control and adaptively move the robot arm, move the ultrasonic probe to the knee joint and its position, move the probe to the dorsal position of the popliteal fossa and the medial position of the midline of the knee joint, scan downward along the blood vessels, and scan the veins of the lower extremities, including : Femoral vein, superficial femoral vein, deep femoral vein, external iliac vein, great saphenous vein, collecting medical images and videos.

A remote and adaptive method for scanning bones, joints, muscles, nerves, collecting bones, organs, and tissue medical images and videos:

S1. According to claim 6, the shoulder, shoulder joint and its position, bone and its position, elbow joint and its position, ankle joint and its position can be intelligently identified through the depth vision acquisition device and multiple medical image fusion intelligent recognition methods .

S2. Through the intelligent recognition method of ultrasonic image and various medical image fusion, the blood vessel color information, blood vessel position information and organ information under the ultrasonic image are intelligently identified for abdominal aorta, groin, superficial femoral artery, popliteal artery, anterior tibial artery, and tibial artery posterior artery.

S3. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the shoulder joint, scan the long head tendon, deltoid muscle, subscapularis tendon and muscle sheath along the shoulder joint toward the arm, and scan the supraspinatus along the shape of the central bone of the shoulder Muscle tendon and infraspinatus tendon, capture medical images and videos.

S4. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the elbow joint, olecranon fossa, along the dorsal side of the elbow joint to scan the head of the columella, in front of the radial head, along the outer side of the elbow joint, and the outer edge of the olecranon Scan the position of the small head of the columella, the rear of the radial head, muscles, nerves, push-pull and flip angle adjustment device to adjust the position of the elbow joint at 90 degrees, scan the inner superior bone and ulna, and collect medical images and videos.

S5. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the knee joint and position, along the outside of the knee joint, scan the side of the patella, scan towards the hand direction to scan the head of the columella, the front of the radial head, along the outside of the elbow joint, and the eagle Scan the head of the cranium and the rear of the radial head at the outer edge of the mouth, and collect medical images and videos.

S6. Remote control and self-adaptive mobile robot arm, ultrasonic probe to ankle joint and position, fibula and talus. The push-pull flip angle adjustment device of the robotic arm assists in bending the ankle joint, scans the anterior talofibular ligament, moves the probe to the round bone surface of the fibula, scans the outer edge of the talus, collects triangular images of the talus, talus, and anterior fibular ligament, and collects medical images and videos .

In summary, the beneficial effects of the present invention

The present invention can use a medical robot device, remotely isolate and autonomously collect, isolate and collect, autonomously locate and identify the characteristic positions of human organs, and collect ultrasonic images. Independently complete outpatient and ward medical care tasks. And in order to improve the problems of many collection tasks, high work pressure, and many night shifts.

At the same time, the present invention provides a remote and self-adaptive method for collecting neck, left and right thyroid lobes, isthmus, parotid gland, images, and videos; a remote and self-adaptive method for collecting blood vessels of the lower extremities, collecting organs and tissues Medical image and video method; a remote and self-adaptive acquisition of bones, joints, muscles, nerve scanning methods, acquisition of organs, medical image organization, video method, to achieve remote and autonomous effective acquisition, complete acquisition of medical images, Video, sharing medical pictures, video, multi-expert remote joint consultation, real-time acquisition of data, images and videos collected by robots, greatly improving work efficiency.

Description of drawings

Fig. 1 is a flow chart of an intelligent recognition method for intelligent recognition of human body features, organs, bones, joints, blood vessels and their positions through the fusion of visual images and medical images in the specification of this application;

Fig. 2 is a flow chart of a method for scanning and collecting medical images and videos by robots and medical devices autonomously positioning the external position area of the human body in the specification of this application;

Detailed ways

The purpose of the present invention is to design a remote-controllable robot that replaces human work, realize remote control of robotic arm acquisition, and effectively solve autonomous image and video acquisition. Using artificial intelligence robot technology, autonomous collection in the field of automation, robot arm motion planning, and depth cameras to collect images of faces, facial features, arms, external features of the human body, bones, and joints.

Fusion of visual images and medical images to intelligently recognize human body features, organs, bones, joints, blood vessels, intelligent recognition methods for human body features and their positions, as well as autonomous positioning of robots, robotic arms, medical devices, external position areas of the human body, scanning and collection of medical images The method becomes autonomous positioning and scanning.

The present invention provides a remote and self-adaptive intelligent recognition, scans the neck, left and right thyroid lobes, isthmus, parotid gland, and collects images and video methods; a remote and self-adaptive intelligent recognition, scans and collects blood vessels of the lower extremities A method for collecting organs, organizing medical images and videos; a method for remote and adaptive intelligent recognition, collecting bones, joints, muscles, and nerve scans, collecting organs, organizing medical images and videos.

Realize remote control of robots and autonomous collection of medical images, videos, remote control of medical images, video collection devices and shared images, solve human errors in diagnosis and treatment, improve the accuracy of intelligent collection and the accuracy of abnormal identification of medical data. In order to better understand the above technical solutions, the present invention will be further described in detail below in conjunction with the examples and accompanying drawings, but the embodiments of the present invention are not limited thereto.

The technical scheme in the implementation of this application is as follows for the overall idea of solving the above-mentioned technical problems:

The invention provides an intelligent recognition method for medical image and visual image fusion, intelligent recognition of human facial features, organs, bones, joints, blood vessels, human body features and their positions, and establishment of general visual image models, human organ models, blood vessel models and feature models, The general image model includes: face, facial features, ears, lips, mandible, neck, navel, nipples, characteristic genitals, characteristic parts and their positions as characteristic items, using neural network algorithms and their improved methods to intelligently identify human bodies features and their location. Establish a depth vision device, a depth vision image model, and a skeleton model. Using the depth information and the position information of the human body where the bones are located, a method of autonomously locating and identifying the characteristic positions of human organs, and collecting ultrasonic images;

And a method for remote and adaptive collection of neck, left and right thyroid lobes, isthmus, parotid gland, images, and videos; a remote and adaptive collection of blood vessels in the lower extremities, collection of organs, tissue medical images, and video Method; a method for remote and self-adaptive collection of bones, joints, muscles, and nerve scans, collection of organs, and medical images and videos.

Example 1:

Below in conjunction with embodiment and accompanying drawing 1, the present invention will be described in further detail, but the embodiment of the present invention is not limited thereto, input general visual image, human body organ, blood vessel and feature, include: human face, facial features, ear, Lips, mandible, neck, navel, nipples, characteristic genitals, characteristic parts and their positions, input depth visual image, bone information, mandible, bones at the base of ribs, xiphoid process, spine, spine position, lower bone position, shoulder Joints, knee joints, feet, bones, foot joints, waist joints and their positions, the position of each joint.

Blood vessel color information, blood vessel position information and ultrasound image organ contour shape features, structural features, and color features under the ultrasound image. Combining blood vessel information, blood vessel position information with organ information and organ features under the ultrasound image into an information combination item as a characteristic item and an input item of the ultrasound image model. Use the external scanning feature information and external scanning bone information as external scanning information, apply the improved deep neural network method and weight optimizer, and obtain output values and organs, blood vessels, bones, and human body scanning where they are located through image training Organ and location information, output the results of autonomous positioning of organs, blood vessels, bones, and their positions in the human body.

According to the administrator, the doctor releases the acquisition task, the doctor's order message, obtains the coordinates of the organ and its external location area of the human body acquisition image corresponding to the acquisition task, sets it as the target, sets the target name, target parameters, location information, and sets the communication target.

The robot vision acquisition device and visual recognition module release the external features corresponding to each organ, and the external scanning feature information of the adjacent characteristic organs recognized by the general image model The coordinates of the external position area of the human body; the depth information released by the depth camera, the external scanning of the adjacent bone information Skeletal information.

Robot arm, ultrasound device, ultrasound probe, and robot subscribe to the location information of the corresponding organ external scan area, subscription target, parameter, target pose, pose mark, set target for head id, target pose, direction value, set time Stamp, the remote main control system and the ultrasonic probe carried by the autonomous robot arm move and scan the human body acquisition area according to the subscribed acquisition area location and the action of the robotic arm image acquisition action planning module. The ultrasonic probe and the ultrasonic device release the collected image information, blood vessel color information, blood vessel position information and contour shape features, structural features, and color features of the organs in the ultrasonic image. Robots, medical devices and visual recognition modules subscribe to image information. Extract blood vessel color information, blood vessel position information and ultrasonic image organ contour shape features, structural features, and color features into the calculation model to intelligently identify whether the image is the target organ tissue, and if it is the target organ for scanning.

Set acquisition target parameters (pose mark, timestamp, target for head id, COG target pose, direction value), set the allowable error of position and attitude, allow re-planning when motion planning fails, and set the reference of target position Coordinate system, set the time limit for each motion planning. The image parameters of the robot, the medical device and the remote end of the robot arm and the self-adaptive adjustment and acquisition of the medical device, the parameters of the video, the image acquisition method, whether it meets the image and video recognition standards, and whether the acquisition is effective. Robots, medical devices, remote ends of robotic arms, and self-adaptive adjustments to the scanning method, probe angle, and parameters of the ultrasound probe, to check the acquisition position of the target organ, and whether all images and videos of the collected organs and tissues are all images in the scanning mode , video, whether it is the target organ, complete collection of tissue, return target organ collection completion information, robot and robot arm subscribe task information, robot, medical device and robot arm distal end and adaptively move the ultrasound probe to the outside of the next target organ Scan the location area. The robot and the medical device determine that all target organ collection tasks are completed according to the returned target organ collection completion information.

Example 2:

Below in conjunction with embodiment and accompanying drawing 1, 2, the present invention is described in further detail, but the embodiment of the present invention is not limited to this kind of far-end and self-adaptive scan neck, thyroid left and right lobes, isthmus, parotid gland, acquisition image, video method,

According to the general visual device and a variety of medical image fusion intelligent recognition methods, intelligently identify the human ear, auricle and its position, intelligently identify the human lip and its position, through the depth vision acquisition device and a variety of medical image fusion intelligent recognition methods , Intelligently identify bones, intelligently identify the mandible at the bottom of the ear, spine, bone position, spine position, intelligently identify bones, and the bone position below the midline of the lip.

According to the autonomous positioning of the robot, the coordinates of the external position area of the human body, the method of scanning and collecting medical images, the robotic arm, ultrasonic device, ultrasonic probe, robot, and medical device subscribe to the position information of the external scanning area of the neck, left and right thyroid lobes, isthmus, and parotid gland, Scanning method, probe angle, subscription target, parameters, target pose, pose marker, set target for head id, target pose, direction value, set timestamp.

Image parameters, gain parameters, color gain parameters, sensitivity time control adjustment parameters, time gain control parameters, focus parameters, depth parameters, frame size, blood flow of robots, medical devices and robotic arms, and self-adaptive adjustment and collection of medical devices Velocity scale parameters, video parameters, image acquisition method, robotic arm, ultrasonic probe, remote control and adaptive adjustment of probe rotation angle, tilt angle, ultrasonic probe scanning method, probe angle, parameters to the following organs, effective collection of tissues , complete collection.

Remotely control and adaptively move the robot arm, scan and collect images and videos of the neck, spine, common carotid artery, internal carotid artery, and external carotid artery along the lower boundary of the bone at the bottom of the ear with the ultrasonic probe.

Remotely control and adaptively move the robotic arm, the ultrasonic probe moves along the bone position below the midline of the lip, and moves along the carotid artery from the head side to the foot side to scan the left and right lobes and isthmus of the thyroid gland, and collect medical images and videos.

Remote control and adaptive mobile robot arm, ultrasonic probe, along the bottom of the ear, along the bone position, along the mandible, bone position, scan the parotid gland, submandibular gland, sublingual gland to the bone position below the midline of the lip, collect medical images and video.

Example 3

Below in combination with the embodiments and accompanying drawings 1 and 2, the present invention will be further described in detail for the far-end and self-adaptive acquisition of scanning methods for blood vessels of the lower extremities, methods for collecting organs, organizing medical images, and videos:

According to the general visual device and a variety of medical image fusion intelligent recognition methods, it can intelligently identify the human ear, auricle and its position, and intelligently identify the navel and its position.

Through the depth vision acquisition device and a variety of medical image fusion intelligent recognition methods, intelligently identify bones, rib bottom bones and xiphoid process positions, ischium, knee joints and their positions, dorsally measure popliteal fossa and their positions, foot bones, and foot joints and its location.

Through the intelligent recognition method of ultrasonic image and various medical image fusion, the blood vessel color information, blood vessel position information and organ information under the ultrasonic image can be intelligently identified abdominal aorta, groin, superficial femoral artery, popliteal artery, anterior tibial artery, posterior tibial artery .

Through the intelligent recognition method of ultrasonic image and a variety of medical image fusion, the dynamic recognition method under the ultrasonic image is used to identify the pulsation position as the groin area.

Remote control and self-adaptive mobile robot arm, ultrasonic probe to the groin area, the ultrasonic probe scans the groin area from the xiphoid process down to the navel and along the abdominal aorta to collect medical images and videos.

Remotely control and adaptively move the robot arm, scan the superficial femoral artery with the ultrasonic probe along the groin area, move to the knee joint and its position, dorsally measure the position of the popliteal fossa, and scan from the dorsal position of the popliteal fossa to the blood vessels near the dorsal side of the knee joint Check and collect medical images and videos.

Remote control and self-adaptive mobile robot arm, ultrasonic probe scans downward along the popliteal artery, anterior tibial artery, and posterior tibial artery, presses with a pressing device, scans nearby blood vessels along the anterior tibial artery from the knee joint, collects medical images and video.

Remote control and self-adaptive mobile robot arm, ultrasonic probe to the foot bone, foot joint and its position, scan the dorsalis pedis artery along the dorsum of the foot, scan the plantar artery along the sole of the foot, and collect medical images and videos.

Remotely control and adaptively move the robot arm, move the ultrasonic probe to the knee joint and its position, move the probe to the position of the dorsal popliteal fossa and the medial position of the midline of the knee joint, scan downward along the blood vessels, and scan the veins of the lower extremities, including: femoral Vein, superficial femoral vein, deep femoral vein, external iliac vein, great saphenous vein, collect medical images and videos.

Example 4

Below in conjunction with the embodiments and accompanying drawings 1 and 2, the present invention will be further described in detail, remote and adaptive collection of bone joints, muscles, nerve scanning methods, collection of organs, tissue medical images, video methods:

Through the deep vision acquisition device and various medical image fusion intelligent recognition methods, it can intelligently recognize shoulders, shoulder joints and their positions, bones and their positions, elbow joints and their positions, ankle joints and their positions.

Remote control and adaptive mobile robotic arm, ultrasonic probe to the shoulder joint, scan the long head tendon, deltoid muscle, subscapularis tendon and muscle sheath along the shoulder joint toward the arm, scan the supraspinatus tendon along the shape of the central bone of the shoulder With the infraspinatus tendon, capture medical images and videos.

Remote control and self-adaptive mobile robot arm, ultrasonic probe to the elbow joint, olecranon fossa, scan the head of the columella along the dorsal side of the elbow joint to the hand direction, scan along the outside of the elbow joint and the outer edge of the olecranon in front of the radial head Check the capitulum, the rear of the radial head, muscles, nerves, push-pull and flip angle adjustment device to adjust the 90-degree position of the elbow joint, scan the inner superior bone and ulna, and collect medical images and videos.

Remote control and self-adaptive mobile robot arm, ultrasonic probe to the knee joint and position, scan along the outside of the knee joint, the side of the patella, scan towards the hand direction, scan the head of the columella, the front of the radial head, along the outside of the elbow joint, and the outside of the olecranon Scan the head of the cranium and the rear of the radial head at the edge position, and collect medical images and videos.

Remote control and adaptive movement of the robotic arm, ultrasound probe to the ankle joint and position, fibula and talus. The push-pull flip angle adjustment device of the robotic arm assists in bending the ankle joint, scans the anterior talofibular ligament, moves the probe to the round bone surface of the fibula, scans the outer edge of the talus, collects triangular images of the talus, talus, and anterior fibular ligament, and collects medical images and videos .

Claims

The visual image and medical image fusion recognition, autonomous positioning scanning method is characterized in that it is an intelligent recognition method that integrates human facial features, organs, bones, joints, blood vessels, human body features and their positions. The method is to combine general visual images, depth A method for intelligently identifying human organs, bones, joints, blood vessels, and human body feature points and their positions through the integration of images, ultrasound images, and endoscopic images, and a method for autonomously positioning human organs, bones, joints, blood vessels, and human body feature points, including The following steps:

S1. Establish general visual image models, human organ models, blood vessel models and feature models. The general image models include: face, facial features, ears, lips, mandible, neck, navel, nipples, characteristic genitalia, characteristic parts and Its position is used as a feature item, and the neural network algorithm and its improved method are used to intelligently identify the human body's features and its position.

S2. Establish a depth vision device, a depth vision image model, and a skeleton model.

S3. Combining the depth information, the position information of the human body where the bones are located, and the information of the adjacent characteristic organs identified by the general image model described in S1 as the feature items of the intelligent bone recognition model as input items.

S4. Apply the neural network algorithm and its improved method to intelligently identify each bone and the position of each bone, mandible, costal bottom bone, xiphoid process, spine, spine position, bone lower boundary position, shoulder joint, knee joint, foot, Bones, foot joints, lumbar joints and their positions, the position of each joint.

S5. Standardize the adjacent characteristic organ information identified by the general image model described in S1 as external scanning feature information, and externally scan the bone information for the bone information identified in S4,

S6. Establish a feature model under the ultrasound image, and combine the color information of blood vessels under the ultrasound image, the location information of blood vessels, and the contour shape features, structural features, and color features of organs in the ultrasound image.

S7. Combining blood vessel information, blood vessel position information with organ information and organ features under the ultrasound image into an information combination item as a feature item and an input item of the ultrasound image model.

S8. Using the external scan feature information and the external scan bone information as the external scan information, combine the adjacent characteristic organ information identified by the general image model described in S1 with the bone information identified by S4 and the features of the ultrasound image model described in S7 The item and its location area are input into the neural network and its improved method and weight optimizer, and the output value is obtained through image training.

S9. Improve the deep neural network method and the weight optimizer, and obtain the output value and the recognition results of the position information of the organs, blood vessels, bones, and the human body where they are located through image training.

S10. Outputting the result, as the result of external scanning to autonomously locate organs, blood vessels, bones, and their positions in the human body.
The visual image and medical image fusion recognition, autonomous positioning scanning method is characterized in that a robot autonomously locates the external position area of the human body, and scans and collects medical images. The method includes the following steps:

S1. According to the administrator, the doctor communication module, and the main robot system publish collection tasks and doctor’s order messages, obtain the coordinates of the organ and its external position area of the human body collection image corresponding to the collection task, set it as the target, set the target name, target parameters, and location information, set communication goals.

S2. The robot vision acquisition device and visual recognition module release the external features corresponding to each organ, the external scanning feature information of the nearby characteristic organs recognized by the general image model The coordinates of the external position area of the human body; the depth information released by the depth camera, the external bone information of the nearby neighbors Scan bone information.

S3. The robot arm, ultrasound device, ultrasound probe, and robot subscribe to the location information of the corresponding organ external scan area, subscribe to the target, parameters, target pose, pose mark, set the target for the head id, target pose, direction value, set timestamp

S4. The remote main control system and the ultrasonic probe mounted on the autonomous robot arm move and scan the human body acquisition area according to the subscribed acquisition area location and the action of the robotic arm image acquisition action planning module. The ultrasonic probe and the ultrasonic device release the collected image information, blood vessel color information, blood vessel position information and contour shape features, structural features, and color features of the organs in the ultrasonic image.

S5. The main robot system and the visual recognition module subscribe to the image information. According to claim 1, the blood vessel color information, blood vessel position information and ultrasonic image organ contour shape features, structural features, and color features are input into the calculation model according to claim 1, and intelligently identify whether the image is the target organ tissue, if it is Scan target organs.

S6. Set the acquisition target parameters (pose mark, time stamp, target for head id, COG target pose, direction value), set the allowable error of position and attitude, when the motion planning fails, allow re-planning, set the target position The reference coordinate system, set the time limit for each motion planning.

S7. The main system of the robot and the remote end of the robot arm and self-adaptive adjustment to collect the image parameters of the medical device, the parameters of the video, the image collection method, whether it meets the image and video recognition standards, and whether the collection is effective.

S8. The main system of the robot and the remote end of the robot arm and self-adaptive adjustment of the scanning mode, probe angle, and parameters of the ultrasonic probe, check the collection position of the target organ, and whether all the images and videos of the collected organs and tissues are in the scanning mode. Images, videos, whether it is the target organ or not, complete collection of tissues.

S9. Return the target organ acquisition completion information, the robot main system and the robot arm subscribe to the task information, the robot main system and the robot arm distal end and adaptively move the ultrasound probe to the external scanning position area of the next target organ.

S10. The main robot system determines that all target organ collection tasks are completed according to the returned target organ collection completion information.
The visual image and medical image fusion recognition, self-positioning scanning method is characterized in that it is a method of remote and adaptive scanning of the neck, left and right thyroid lobes, isthmus, and parotid gland, and collecting medical images and videos. The steps are as follows:

S1. According to claim 1, according to the general visual device and a variety of medical image fusion intelligent recognition methods, intelligent recognition of human ears, auricles and their positions, intelligent recognition of human lips and their positions, through the depth vision acquisition device and multiple A medical image fusion intelligent identification method, intelligent identification of bones, intelligent identification of the mandible at the bottom of the ear, spine, bone position, spine position, intelligent identification of bones, and the position of bones below the midline of the lip.

S2. According to claim 2, according to the autonomous positioning of the robot, the coordinates of the external position area of the human body, the method of scanning and collecting medical images, the robot arm, the ultrasonic device, the ultrasonic probe, the main system of the robot subscribes to the neck, the left and right thyroid lobes, the isthmus, and the outside of the parotid gland Location information of the scanning area, scanning method, probe angle, subscription target, parameters, target pose, pose mark, set target for head id, target pose, direction value, set timestamp.

S3. The main system of the robot and the remote end of the robot arm and self-adaptive adjustment and acquisition of image parameters, gain parameters, color gain parameters, sensitivity time control adjustment parameters, time gain control parameters, focus parameters, depth parameters, frame size, blood Flow rate scale parameters, video parameters, image acquisition method, robotic arm, ultrasonic probe, remote control and adaptive adjustment of probe rotation angle, tilt angle, ultrasonic probe scanning method, probe angle, parameters are valid for the following organs and tissues Collection, complete collection.

S4. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans along the lower boundary of the bone at the bottom of the ear, and collects images and videos of the neck, spine, common carotid artery, internal carotid artery, and external carotid artery.

S5. Remote control and self-adaptive mobile robot arm, the ultrasonic probe moves along the bone position below the midline of the lip, along the carotid artery from the head side to the foot side to scan the left and right lobes and isthmus of the thyroid gland, and collect medical images and videos.

S6. Remote control and adaptive mobile robot arm, ultrasonic probe, along the bottom of the ear, bone position, along the mandible, bone position, scan the parotid gland, submandibular gland, sublingual gland to the bone position below the midline of the lip, collect medical treatment images and videos.
The visual image and medical image fusion recognition, self-positioning scanning method is characterized in that it is a method of remote and adaptive scanning of blood vessels of lower limbs, collecting blood vessels of lower limbs, organs, medical images of tissues, and video. The steps are as follows:

S1. According to claim 1, according to the general visual device and multiple medical image fusion intelligent recognition methods, intelligently identify human ears, auricles and their positions, and intelligently identify navel and their positions.

S2. Through the deep vision acquisition device and various medical image fusion intelligent recognition methods, intelligently identify bones, rib bottom bones and xiphoid process positions, ischium, knee joints and their positions, dorsally measure popliteal fossa and its positions, foot bones, Foot joints and their positions.

S3. According to claim 1, through the ultrasonic image and multiple medical image fusion intelligent recognition method, the blood vessel color information, blood vessel position information and organ information under the ultrasonic image are intelligently identified for abdominal aorta, groin, superficial femoral artery, popliteal artery, Anterior tibial artery, posterior tibial artery.

S4. According to claim 1, through the intelligent identification method of fusion of ultrasonic images and multiple medical images, the dynamic identification method under ultrasonic images is used to identify the pulsating position as the groin area.

S5. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the groin area, the ultrasonic probe scans the groin area from the position of the xiphoid process down to the navel and along the abdominal aorta to collect medical images and videos.

S6. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans the superficial femoral artery along the groin area, moves to the knee joint and its position, measures the position of the popliteal fossa from the back, and moves from the position of the popliteal fossa to the dorsal side of the knee joint Vascular scanning, collecting medical images and videos.

S7. Remote control and self-adaptive mobile robot arm, the ultrasonic probe scans downward along the popliteal artery, anterior tibial artery, and posterior tibial artery, presses with a pressing device, scans nearby blood vessels along the anterior tibial artery from the knee joint, and collects medical records. images and videos.

S8. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the foot bones, foot joints and their positions, scan the dorsalis pedis artery along the dorsum of the foot, scan the plantar artery along the sole of the foot, and collect medical images and videos.

S9. Remotely control and adaptively move the robot arm, move the ultrasonic probe to the knee joint and its position, move the probe to the dorsal position of the popliteal fossa and the medial position of the midline of the knee joint, scan downward along the blood vessels, and scan the veins of the lower extremities, including : Femoral vein, superficial femoral vein, deep femoral vein, external iliac vein, great saphenous vein, collecting medical images and videos.
The visual image and medical image fusion recognition, autonomous positioning scanning method, is characterized in that it is a remote and adaptive scanning method for bones, joints, muscles, nerves, collection of bones, organs, tissue medical images, and videos, the steps are as follows :

S1. According to claim 1, the shoulder, shoulder joint and its position, bone and its position, elbow joint and its position, ankle joint and its position can be intelligently identified through the depth vision acquisition device and multiple medical image fusion intelligent recognition methods .

S2. Through the intelligent recognition method of ultrasonic image and various medical image fusion, the blood vessel color information, blood vessel position information and organ information under the ultrasonic image are intelligently identified for abdominal aorta, groin, superficial femoral artery, popliteal artery, anterior tibial artery, and tibial artery posterior artery.

S3. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the shoulder joint, scan the long head tendon, deltoid muscle, subscapularis tendon and muscle sheath along the shoulder joint toward the arm, and scan the supraspinatus along the shape of the central bone of the shoulder Muscle tendon and infraspinatus tendon, capture medical images and videos.

S4. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the elbow joint, olecranon fossa, along the dorsal side of the elbow joint to scan the head of the columella, in front of the radial head, along the outer side of the elbow joint, and the outer edge of the olecranon Scan the position of the small head of the columella, the rear of the radial head, muscles, nerves, push-pull and flip angle adjustment device to adjust the position of the elbow joint at 90 degrees, scan the inner superior bone and ulna, and collect medical images and videos.

S5. Remote control and self-adaptive mobile robot arm, ultrasonic probe to the knee joint and position, along the outside of the knee joint, scan the side of the patella, scan towards the hand direction to scan the head of the columella, the front of the radial head, along the outside of the elbow joint, and the eagle Scan the head of the cranium and the rear of the radial head at the outer edge of the mouth, and collect medical images and videos.

S6. Remote control and self-adaptive mobile robot arm, ultrasonic probe to ankle joint and position, fibula and talus. The push-pull flip angle adjustment device of the robotic arm assists in bending the ankle joint, scans the anterior talofibular ligament, moves the probe to the round bone surface of the fibula, scans the outer edge of the talus, collects triangular images of the talus, talus, and anterior fibular ligament, and collects medical images and videos .
The visual image and medical image fusion recognition, autonomous positioning scanning method is characterized in that the visual image and medical image fusion recognition, and the autonomous positioning scanning method is suitable for remote and autonomous positioning scanning devices, which include robots, robotic arms, Medical devices, medical device control devices, ultrasonic devices, ultrasonic probe devices include one or more of scanning devices and scanning control devices, one or more of planar vision devices, and one or more of depth vision devices. species and varieties.