WO2023274270A1

WO2023274270A1 - Robot preoperative navigation method and system, storage medium, and computer device

Info

Publication number: WO2023274270A1
Application number: PCT/CN2022/102141
Authority: WO
Inventors: 姬亚楠; 程陈; 何超
Original assignee: 上海微觅医疗器械有限公司
Priority date: 2021-06-30
Filing date: 2022-06-29
Publication date: 2023-01-05
Also published as: CN115542889A

Abstract

A surgical robot (11) preoperative navigation method, a robot preoperative navigation system, a storage medium, and a computer device. The preoperative navigation method comprises: acquiring environmental information in an operating room, and displaying a virtual map on the basis of the environmental information, a mark of the starting position of the surgical robot (11) and a mark of a surgical operation position being formed in the virtual map (S11); and acquiring first interaction information, and generating a planned path of the surgical robot (11) in the virtual map on the basis of the first interaction information, the planned path being used for the surgical robot (11) to move, in the operating room, from the starting position to the surgical operation position correspondingly (S12). The surgical robot (11) preoperative navigation method, the robot preoperative navigation system, the storage medium, and the computer device can be applied to navigation in the operating room.

Description

Robot preoperative navigation method, system, storage medium and computer equipment

This application claims the priority of the Chinese patent application submitted to the China Patent Office on June 30, 2021, with the application number 2021107450216, and the application name is "robot preoperative navigation method, system, storage medium and computer equipment", the entire content of which is passed References are incorporated in this application.

technical field

The present application relates to the technical field of medical robots, in particular to a robot preoperative navigation method, a robot preoperative navigation system, storage media and computer equipment.

Background technique

In traditional technology, the Global Positioning System (GPS, Global Positioning System) sensor is used to locate the starting position and target position of the robot and the path between them. This method plans the navigation path of the robot by matching the latitude and longitude coordinates. However, due to the weak indoor GPS signal, this method is not suitable for the navigation of the surgical robot in the operating room, so it cannot realize the effective obstacle avoidance of the surgical robot moving from the initial position to the operating position in the operating room.

Contents of the invention

Based on this, it is necessary to provide a robot preoperative navigation method, a robot preoperative navigation system, a storage medium and a computer device for the above problems.

A preoperative navigation method for a robot, comprising: obtaining environmental information in an operating room, and displaying a virtual map based on the environmental information; a mark of a starting position of a surgical robot and a mark of a surgical operation position are formed in the virtual map; and Acquiring first interaction information, and generating a planned path of the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to correspondingly start in the operating room The position moves to the surgical operation position.

In one of the embodiments, the virtual map is superimposed and displayed in the real scene.

In one of the embodiments, the virtual map includes a grid mark, and the mark of the starting position and the mark of the operation operation position are located at the intersection of the mark of the grid; the planned path passes through the The intersection marked as no obstacle in the grid mark connects the mark of the starting position and the mark of the surgical operation position.

In one of the embodiments, the acquiring the environmental information in the operating room and displaying the virtual map based on the environmental information includes: obtaining the environmental information by using multiple pixel images taken at different positions in the operating room; Constructing and displaying a virtual map with a grid mark according to the environmental information; wherein, some intersection points in the grid mark display the environmental information.

In one of the embodiments, the environmental information includes position coordinate information of obstacles in the operating room and the starting position; the construction and display of a virtual map with a grid mark according to the environmental information includes: Based on the position coordinate information of the marked position of the surgical operation position, and the position coordinate information of the obstacle and the initial position, expand the coordinate information of a plurality of virtual points that do not overlap with each position coordinate information; The coordinate information of the virtual point and each location coordinate information are intersection points of the grid mark, and the virtual map with the grid mark is generated and displayed.

In one of the embodiments, the shape of the grid in the virtual map with the grid mark includes a triangle.

In one of the embodiments, the generating the planned path of the surgical robot in the virtual map based on the first interaction information includes: determining the path relative to the starting position based on the first interaction information. at least one pose change information to obtain a planned path of the surgical robot; and display the planned path on the virtual map.

In one of the embodiments, during the movement of the surgical robot from the initial position to the surgical operation position, the method further includes: acquiring second interaction information, and adjusting The motion state of the surgical robot, and/or giving prompt information.

In one embodiment, the robot preoperative navigation method further includes: projecting the planned path in the operating room between the starting position and the surgical operation position according to a corresponding scale.

In one embodiment, the need to adjust the motion state during the movement of the surgical robot includes: the distance between the surgical robot and obstacles is less than a preset safety distance and/or the trajectory of the surgical robot deviates from the specified Describe the planned path.

In one of the embodiments, the method further includes: controlling the movement of the surgical robot and/or controlling the position of the mechanical arm of the surgical robot based on the planned path.

A robot preoperative navigation system, comprising: a control processing device and a human-computer interaction device, the control processing device and the human-computer interaction device are connected in communication; the human-computer interaction device is used to display a virtual map, and is used to obtain user The first interaction information; the control processing device is used to execute the robot preoperative navigation method described in any one of the above.

In one of the embodiments, the human-computer interaction device includes an AR device, and the AR device is configured to superimpose and display the virtual map in a real scene.

In one of the embodiments, the control processing device includes a position adjustment unit and a camera, the camera is arranged on the position adjustment unit, the position adjustment unit is used to adjust the shooting angle of the camera, and the camera uses Multiple pixel images taken at different positions in the operating room are collected to obtain the environmental information.

In one of the embodiments, the human-computer interaction device projects the planned path in the operating room between the starting position and the surgical operation position according to a corresponding scale.

In one embodiment, during the movement of the surgical robot from the initial position to the surgical operation position in the operating room; the control processing device obtains the second interaction information based on the gesture information, and based on the The second interaction information adjusts the motion state of the surgical robot, and/or gives prompt information.

In one of the embodiments, the human-computer interaction device acquires the first interaction information by collecting gesture images.

A storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

A computer device, including a memory and a processor; the processor stores a computer program that can run on the processor, and the processor implements the steps of any one of the above methods when executing the computer program .

The above robot preoperative navigation method, robot preoperative navigation system, storage medium, and computer equipment obtain environmental information in the operating room to display a virtual map in the operating room. The operator uses gestures to plan the navigation path in advance on the virtual map, and the surgical robot follows the planned path. The motion can automatically and quickly move in place and avoid indoor obstacles, thereby reducing robot collisions and reducing the robot damage rate; and this method can be applied to indoor navigation.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the traditional technology. Obviously, the accompanying drawings in the following description are only the present invention For some embodiments of the application, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flowchart of the robot preoperative navigation method provided in an embodiment;

Fig. 2 is a schematic diagram of a robot system provided in an embodiment;

Fig. 3 is a schematic diagram of the preoperative movement process of the surgical robot provided in one embodiment;

FIG. 4 is a flow chart of specific steps of step S11 provided in an embodiment;

Fig. 5 is a schematic structural diagram of a control processing device provided in an embodiment;

FIG. 6 is a flow chart of specific steps of step S112 provided in an embodiment;

7a to 7d are schematic diagrams of the principles of the gridded navigation path planning method provided in an embodiment;

FIG. 8 is a flow chart of specific steps of step S12 provided in an embodiment;

Fig. 9a is a schematic structural diagram of AR glasses with a binocular camera provided in an embodiment;

Fig. 9b is a schematic diagram of the relative relationship between the coordinate system of the AR glasses and the binocular camera provided in an embodiment;

Fig. 9c is a schematic diagram of the principle of binocular vision provided in an embodiment;

FIG. 10 is a flow chart of specific steps of step S122 provided in an embodiment;

Fig. 11a is a schematic diagram illustrating an implementation manner of histogram-based segmentation provided in an embodiment;

Fig. 11b is a schematic diagram illustrating the implementation of segmentation based on local area information provided in an embodiment;

Fig. 12 is a flow chart of steps further included in the robot preoperative navigation method provided in an embodiment;

Fig. 13 is a schematic diagram of an operator gesture to adjust the motion state of the robot provided in an embodiment;

Fig. 14 is a schematic diagram of the principle of driving forward provided in an embodiment;

Fig. 15 is a structural block diagram of a robot preoperative navigation system provided in an embodiment.

detailed description

In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. A preferred embodiment of the application is shown in the drawings. However, the present application can be embodied in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive.

Fig. 1 is a flow chart of a robot preoperative navigation method in an embodiment. Referring to Figure 1, the robotic preoperative navigation method consists of the following steps:

Step S11 , acquiring environmental information in the operating room, and displaying a virtual map based on the environmental information; the virtual map is formed with a mark of the starting position of the surgical robot and a mark of the surgical operation position.

Specifically, the robot preoperative navigation method can be implemented by using a robot preoperative navigation system, which is used to navigate the preoperative movement of the surgical robot in the robot system in the operating room. Referring to Figure 2, the robotic system is located in the operating room. The robot system may include a doctor operating terminal 12, a patient operating terminal (ie, a surgical robot) 11, a vision platform 13, and surgical instruments 14, among others. In addition to the robot system, the operating room may also include a patient table 15 and a patient on the patient table 15 . Wherein, the mechanical arm of the surgical robot 11 can be used to connect the surgical instrument 14, so that the surgical robot 11 can be used to assist the doctor in the operation.

The environment information in the operating room may include position coordinate information of obstacles in the operating room and starting positions. In the operating room, objects and people other than the surgical robot 11 can be referred to as obstacles in the preoperative movement of the surgical robot 11. The starting position may be the current position of the surgical robot 11 . The surgical operation location may be the location of the patient's lesion.

The robot preoperative navigation system may include a control processing device and a human-computer interaction device. The control and processing device is used to obtain environmental information in the operating room, and reconstruct a virtual map in the operating room based on the environmental information. The virtual map can be a planar image or a stereoscopic image. The obstacles and the surgical robot 11 in the virtual map may be similar to the corresponding real objects, or may be replaced by corresponding symbols. The obstacles in the virtual map may be called obstacle marks, and the obstacle marks correspond to the obstacles in the operating room in the real scene. The surgical robot 11 in the virtual map may be called a surgical robot mark, and the surgical robot mark corresponds to the surgical robot 11 in the operating room of the real scene. Of course, it is also possible not to mark the surgical robot mark on the virtual map, but directly replace it with the mark of the initial position. The distance between each obstacle in the virtual map and the surgical robot 11 can be reduced in direct proportion to the corresponding actual distance.

The mark of the starting position of the surgical robot 11 and the mark of the surgical operation position are formed in the virtual map. In the present application, the starting position in the operating room corresponds to the mark of the starting position on the virtual map, and the operation operation position in the operating room corresponds to the mark of the operation operation position on the virtual map.

The human-computer interaction device may be connected in communication with the control processing device, and the human-computer interaction device acquires and displays a virtual map from the control processing device, and detects the first interaction information input by the user. Wherein, the human-computer interaction device is configured with, for example, a monocular/binocular camera, a touch screen, or a keyboard and mouse, and the first interaction information is obtained by using the human-computer interaction device. Wherein, the first interaction information includes setting at least one of the following information on the virtual map: a mark of the surgical operation location, a planned route, a posture at the surgical operation location, and the like. For example, the first interaction information is determined based on at least one image containing human body gestures, or determined by detecting at least one of the user's swipe operation, click operation, and press operation on the human-computer interaction device . For example, the human body gestures include: hand gestures, eyeball gestures, or lower limb gestures. The human-computer interaction device detects at least one image containing the posture of the human body to obtain the mark of the surgical operation position corresponding to the posture of the human body and the posture after reaching the surgical operation position. As another example, the human-computer interaction device determines the planned route and surgical operation position by detecting the track drawn from the starting position mark on the touch screen and the end point of the track; and by detecting the gesture displayed at the end position on the touch screen option to determine the posture of the surgical robot 11 at the surgical operation position.

Step S12, acquiring the first interaction information, and generating a planned path of the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to move from the initial position to the surgical operation position in the operating room.

Specifically, please refer to FIG. 3 . Before the operation, due to the need for instrument preparation and patient pushing, the surgical robot 11 will be far away from the patient table 15 . After a series of preparatory work is completed, the surgical robot 11 needs to move to a position relatively close to the patient table 15, and use the robotic arm to control the surgical instrument 14. The robotic arm can also control laparoscope and other medical imaging equipment to assist the doctor to complete the operation process smoothly. Therefore, before the operation, the surgical robot 11 needs to be moved to a more suitable position around the patient for positioning of the mechanical arm. In this embodiment, the navigation path is planned in advance before the surgical robot 11 is moved, and the surgical robot 11 can be controlled to move automatically according to the planned path to place the robotic arm at the correct surgical position and posture.

According to the displayed virtual map, the operator can use gestures to connect the mark of the starting position and the mark of the surgical operation position, and avoid obstacles on the virtual map during the connection process.

The control processing device generates the planned path of the surgical robot 11 in the virtual map based on the first interaction information. The human-computer interaction device can display the planned path on the virtual map, and the planned path connects the mark of the starting position and the mark of the surgical operation position on the virtual map.

A driver of the robot may be integrated inside the surgical robot 11 . The control processing device can transmit the planned path to the driver of the surgical robot 11 through wireless communication technologies such as Bluetooth or mobile hotspot (Wi-Fi), so that the driver of the surgical robot 11 can control the surgical robot 11 from the starting point in the real scene according to the planned path. The position moves autonomously to the surgical operation position.

The above robot preoperative navigation method obtains the environmental information in the operating room to display the virtual map of the operating room. The operator uses gestures to plan the navigation path in advance on the virtual map, and the surgical robot 11 can automatically move quickly and avoid the operating room according to the planned path. obstacles, thereby reducing the collision of the surgical robot 11 and reducing the damage rate of the surgical robot 11. This method can be applied to navigation in the operating room. In addition, in the operating room, the patient operation end is integrated on the surgical robot 11, and the planned path is used to control the surgical robot 11 to automatically move from the initial position in the real scene to the surgical operation position, without manpower pushing the patient operation end, thereby reducing manual operations Cost, save time and effort.

In some examples, the virtual map can be overlaid and displayed in the real scene, so that it is convenient for the operator to plan a navigation route on the virtual map through gestures. In other examples, a virtual map may also be displayed through a display screen or the like.

Specifically, the human-computer interaction device may include an Augmented Reality (AR, Augmented Reality) device. The operator can wear AR devices such as AR glasses. The AR device can communicate with the control processing device, and the control processing device can transmit the virtual map to the AR device, so that the virtual map can be superimposed and displayed in the real scene through the AR device.

In some examples, the virtual map includes grid markers. The mark of the starting position, the mark of the operation position and each obstacle can be located at different intersections of the grid mark; or it can also be configured such that the mark of the start position and the mark of the operation position are located at different intersections of the grid mark Each obstacle is distributed on the grid. The operator can connect the mark of the starting position and the mark of the surgical operation position through the intersection marked as no obstacle in the grid mark on the virtual map to form a planned path, so that the surgical robot 11 can move according to the planned path in the operating room. Avoid obstacles in the operating room.

In some examples, referring to FIG. 4 , step S11 may specifically include steps S111 to S112.

Step S111, using multiple pixel images taken at different positions in the operating room to obtain environmental information.

Specifically, referring to FIG. 5 , the control processing device may include a camera 311 and a position adjustment unit 312 . The camera 311 may be disposed on a position adjustment unit 312 , and the position adjustment unit 312 is used to adjust the shooting angle of the camera 311 . Through the multi-degree-of-freedom movement of the position adjustment unit 312, the camera 311 can realize 360-degree rotation and scan the environment in the operating room so as to collect more complete pixel images in the operating room. The control processing device obtains multiple pixel images taken at different positions in the operating room through the camera 311, and environmental information can be obtained according to these pixel images. The control processing device can directly identify the surgical robot 11 , patient lesions and other obstacles in multiple pixel images, so as to obtain the position coordinate information of the starting position, the coordinate information of the surgical operation position and the position coordinate information of obstacles respectively. It can also be configured that the operator determines the coordinate information of the surgical operation position according to these pixel images, and inputs it to the control processing device through the input device. In other examples, the control processing device may include a depth data measuring device, which can directly measure the position coordinates of each obstacle in the operating room, the current position coordinates of the surgical robot, and the position coordinates of the patient's lesion, so that the measured data can be Get environmental information.

Step S112, constructing a virtual map with grid marks according to the environment information; wherein, some intersections in the grid marks display the environment information.

Optionally, the shape of the grid in the grid identifier can be set according to actual requirements. For example, the shape of the grid in the virtual map with the grid mark includes a triangle.

In some examples, referring to FIG. 6 , step S112 may specifically include steps S1121 to S1122.

Step S1121, based on the position coordinate information of the surgical operation position and the position coordinate information of the obstacle and the initial position, expand the coordinate information of multiple virtual points that do not overlap with each position coordinate information.

Step S1122, using the coordinate information of each virtual point and each location coordinate information as intersection points marked by grids to generate a virtual map with grid marks.

Specifically, the control processing device may establish an initial virtual map of the operating room according to the environment information. The initial virtual map may include images reduced in proportion to the surgical robot 11 and each obstacle, and the distance between the surgical robot 11 and each obstacle may also be reduced in proportion. Certainly, the surgical robot 11 and various obstacles in the initial virtual map can also be replaced by corresponding symbols. The initial virtual map does not contain grid marks. Then, the control processing device can establish a grid mark according to the environmental information, and then integrate the grid mark into the initial virtual map, so that the mark of the initial position on the initial virtual map, the mark of the surgical operation position and each obstacle correspond to the grid mark respectively. The intersections of the overlap to form a virtual map with grid identification.

Here, the method of forming the grid mark is described by taking the shape of the grid in the grid mark as a triangle as an example. Referring to Fig. 7a, the coordinate information of the extended virtual point can be calculated based on the position coordinate information of the surgical operation position and the position coordinate information of the obstacle and the starting position and using the triangulation (Delaunay) algorithm of the point set. A discrete point set is generated according to the coordinate information of the virtual point and the coordinate information of each position. Please refer to Fig. 7b, using the Delaunay algorithm to generate a triangular mesh image, that is, a mesh mark, from a set of discrete points. Further, please refer to FIG. 7 c , the grid identification can also be simplified to reduce the calculation amount and improve the generation efficiency of the subsequent planning path. It should be noted that the discrete points corresponding to the starting position and the surgical operation position in the grid mark before simplification and the grid mark after simplification are all at the intersection points of the grid.

The following describes in detail the specific implementation process of the Delaunay algorithm using discrete point sets to generate grid labels:

A. Determine point p3: Suppose there are two points p1 and p2. We call p3 the visible point of the straight line p1p2, and determine p3 according to the following three conditions: (1) p3 is on the right side of side p1p2 (vertex order is clockwise); (2) p3 and p1 are visible, that is, side p1p3 is not Intersect with any constraint edge; (3) p3 and p2 are visible.

B. Determine the DT point: In a constrained Delaunay triangle, the vertex opposite to a side is called the DT point of the side. The process of determining the DT point is as follows:

Step1. Construct the circumcircle C (p1, p2, p3) of Δp1p2p3 and its grid bounding box B (C (p1, p2, p3));

Step2. Visit each grid unit in the grid bounding box in turn: search for unvisited grid units and mark them as currently visited grid units. If there is a visible point p in a certain grid unit, and ∠p1pp2>∠p1p3p2, set p3=p1, and go to Step1; otherwise, go to Step3.

Step3. If all grid units in the current grid bounding box have been marked as currently visited grid units, that is, there are no visible points in C(p1, p2, p3), then p3 is the DT point of p1p2.

C. Algorithm design:

Step1. Take any outer boundary edge p1p2.

Step2. Calculate the DT point p3 to form a constrained Delaunay triangle Δp1p2p3.

Step3. If the newly generated edge p1p3 is not a constraint edge, if it is already in the stack, delete it from it; otherwise, put it into the stack; similarly, p3p2 can be processed.

Step4. If the stack is not empty, take an edge from it and go to Step3; otherwise, the algorithm stops.

Then, integrate the grid logo before or after simplification into the initial virtual map to obtain a virtual map with grid logo.

In other examples, the grid identifier may also be directly constructed according to each location coordinate in the environment information. The grid spacing can be adjusted according to the interval between the coordinates of each position. In this way, a virtual map with grid marks can also be obtained without expanding the virtual points.

In some examples, referring to FIG. 8 , step S12 specifically includes steps S121 to S124.

Step S121, acquiring gesture images.

Step S122, obtaining first interaction information based on the gesture image.

Step S123, based on the first interaction information, determine at least one pose change information relative to the starting position, so as to obtain a planned path of the surgical robot.

Step S124, displaying the planned route on the virtual map.

Specifically, please refer to Figure 7d. After the grid mark is integrated into the initial virtual map for display, the operator can use gestures to start from the starting position on the virtual map and connect the points that want to be connected in turn (the bold line in Figure 7d It is the planned route formed by the operator connecting the desired points through gestures, point A corresponds to the starting position, and point B corresponds to the surgical operation position).

The first interaction information is information about the operator's hand posture. Please refer to FIG. 9a, the upper end of the AR device 20 can be connected to the embedded binocular camera 21 through a printed circuit board (PCB, Printed Circuit Board) to collect the gesture image of the operator, and transmit the gesture image to the control processing device. The control processing device obtains the first interaction information according to the gesture image, and determines at least one pose change information relative to the starting position, so as to obtain the planned path of the surgical robot. For example, when the planned path of the surgical robot 11 is a straight line, it is only necessary to determine a pose change information of the surgical robot 11 relative to the starting position; Multiple pose change information in . The control processing device can record all coordinates and routes on the planned route, and transmit the recorded data to the human-computer interaction device, so that the human-computer interaction device can superimpose the planned route on the corresponding position on the virtual image for display.

Among them, the working principle of the binocular camera in Figure 9a is as follows:

Referring to FIGS. 9b and 9c , the relative coordinate relationship between the AR device 20 and the binocular camera 21 is fixed. The camera coordinate system (X ₅ , Y ₅ , Z ₅ ) can establish a mapping relationship through the mechanical position and the display coordinate system (X ₃ , Y ₃ , Z ₃ ). The camera coordinate system (X ₅ , Y ₅ , Z ₅ ) and the world coordinate system (X ₀ , Y ₀ , Z ₀ ) can establish a mapping relationship through the rotation matrix R and the translation vector t, and the mapping relationship is shown in formula (1) .

Among them, (x _c , y _c , z _c ) is the coordinate value of point P in the camera coordinate system, and (x _w , y _w , z _w ) is the coordinate value of point P in the world coordinate system.

Among them, according to the geometric relationship in Figure 9c, it can be obtained that point P satisfies the formulas (2) to (5)

Wherein, the binocular camera 21 includes a left camera and a right camera, the distance between the left camera and the right camera is b, the distance between the left camera and the right camera to the x axis is f, and the point P(x, y, z) The distance from the intersection of the line connecting the left camera and the x-axis to the z-axis is x _l , the distance from the intersection point of the line connecting the left camera and the x-axis to the y-axis is y _l , and the distance from the point P(x, y, z) to the x-axis is y l The distance between the intersection of the line connecting the camera and point P and the x-axis to the line where the right camera is located and parallel to the z-axis is x _r , and the distance between point P and the line where the right camera is located and parallel to the z-axis is ( xb).

In some examples, referring to FIG. 10 , step S122 specifically includes steps S1221 to S1224.

Step S1221, preprocessing the gesture image to obtain a gesture contour image.

Step S1222, extracting geometric moment features of the gesture contour image.

Step S1223, based on the geometric moment feature of the gesture contour image, calculate the distance between the gesture images at different angles at the same moment.

Step S1224, based on the distance between the gesture images at different angles at the same moment, the gesture at that moment is recognized, so as to obtain first interaction information.

Specifically, a recognition algorithm of geometric moments and edge detection may be used for preprocessing. First, binarize the gesture image to obtain the gesture contour image. Then extract the geometric moment feature of the gesture contour image. Specifically, four components of the seven vectors can be taken out, the edge of the image can be directly detected on the basis of the grayscale image, and the boundary direction feature of the image can be represented by a histogram. Finally, the distance between images is calculated by setting the weight of the geometric moment feature, and then the gesture is recognized. This method uses two or more cameras (in this embodiment, the left camera and the right camera in the binocular camera 21) to simultaneously acquire images, as if humans observe the world with their eyes and insects with multi-eye compound eyes. By comparing the difference between the images obtained by these different cameras at the same time, an algorithm is used to calculate the depth information, so as to achieve multi-view 3D imaging.

In some examples, the preprocessing method may specifically include performing any one of histogram-based segmentation, local area information-based segmentation, and physical feature-based segmentation on the gesture image. The three pretreatment methods are illustrated as examples below.

Please refer to Figure 11a. In the histogram-based segmentation, the peak-valley structure can be well determined through the preprocessing and contour tracking of the histogram, so as to find a reasonable segmentation threshold. As long as there is a multi-peak structure in the image histogram and an ideal segmentation threshold, this method has a good segmentation effect.

Please refer to Fig. 11b. In the segmentation based on local area information, the contour extraction can generally obtain the coordinate information of the boundary points through the method of edge detection. Typical is the eight-neighborhood search algorithm to extract the coordinates of gesture boundary points. Each point has eight adjacent points. If one of the points is used as the starting boundary point, the next boundary point must be within the eight neighbors of the point. A closed contour map can be extracted through algorithm tracking .

In the segmentation based on physical features such as color, the skin color is extracted through the YCbCr color space and the skin color modeling based on the Gaussian model, and the motion information analysis is performed through the image difference operation to remove the skin-like background in the image. The accuracy of lower gesture segmentation.

In some examples, please refer to FIG. 12 , the robot preoperative navigation method further includes steps S13 to S16.

Step S13, control the surgical robot to move from the initial position to the surgical operation position in the operating room based on the planned path.

Specifically, the driver of the surgical robot 11 can control the movement of the surgical robot 11 based on the planned path and/or control the positioning of the robotic arm of the surgical robot 11, so that the control center of the robotic arm of the surgical robot 11 can change from the current position in the operating room to Move to the surgical operating position.

Step S14, judging whether the movement state of the surgical robot needs to be adjusted during the movement process.

Specifically, the preoperative navigation system of the robot can automatically determine whether the movement state of the surgical robot 11 needs to be adjusted, and when it is determined that the movement state needs to be adjusted, a prompt can be sent to the operator, so that the operator can perform step S15. It is also possible for the operator to judge whether the motion state needs to be adjusted during the motion of the surgical robot 11 , and step S15 can be executed when it is judged that the motion state needs to be adjusted. The need to adjust the motion state during the movement of the surgical robot 11 may include that the distance between the surgical robot 11 and obstacles is less than a preset safety distance and/or the trajectory of the surgical robot 11 deviates from the planned path, and the like. If it is determined that the movement state of the surgical robot 11 does not need to be adjusted during the movement, step S16 may be performed, and the surgical robot 11 continues to move along the planned path until reaching the surgical operation position.

Step S15, acquiring second interaction information, and adjusting the motion state of the surgical robot based on the second interaction information, and/or giving prompt information. Wherein, the second interaction information is information detected by the human-computer interaction device and used for adjusting a part of the path of the surgical robot 11 during the movement of the surgical robot 11 . The same as or similar to the manner of obtaining the first interaction information, in order to distinguish the first interaction information, in some examples, the second interaction information and the first interaction information are in different working modes of the surgical robot 11 . For example, when the surgical robot 11 works in the stop mode, it corresponds to acquiring the first interaction information, and when it works in the moving mode, it corresponds to acquiring the second interaction information. In some other examples, the second interaction information and the first interaction information are information represented by different detection signals or different image features. For example, the second interaction information is at least one image containing a left-turn (or right-turn) gesture. As another example, the second interaction information is information generated by clicking a turn left (or turn right) button.

Specifically, referring to FIGS. 13 and 9 a , when it is determined that the movement state of the surgical robot 11 needs to be adjusted during the movement, the operator issues an adjustment gesture. A binocular camera 21 (ie, a binocular vision module) can be set on the AR device 20 worn by the operator to collect images of the operator's adjustment gestures, and transmit the adjustment gesture images to the control processing device. After acquiring the adjustment gesture image, the control processing device may acquire second interaction information according to the adjustment gesture image, and adjust the motion state of the surgical robot 11 based on the second interaction information. In other examples, the control processing device may give prompt information based on the second interaction information, for example, when the distance between the surgical robot 11 and the obstacle is less than a preset safety distance, the control processing device may issue a voice prompt. In other examples, steps S13 and S14 may not be executed by the control processing device, and the control processing device executes step S15 during the movement of the surgical robot 11 from the initial position to the surgical operation position.

In some examples, when the robotic preoperative navigation system judges in step S14 whether the surgical robot 11 needs to adjust its motion state during the motion, the robotic preoperative navigation method may also include: acquiring the position information of the

surgical robot

11, and 11 to determine whether the motion state of the surgical robot 11 needs to be adjusted; if the motion state of the surgical robot 11 needs to be adjusted during the motion, a prompt message is output.

Specifically, the position information of the surgical robot 11 may include the distance between the obstacle closest to the surgical robot 11 during the real-time movement of the surgical robot 11 . A distance measuring device such as an ultrasonic distance measuring device can be installed on the surgical robot 11 to obtain the position information of the surgical robot 11 , and transmit the position information of the surgical robot 11 to the control processing device. After the control processing device acquires the position information of the surgical robot 11, it judges whether the motion state of the surgical robot 11 needs to be adjusted based on the position information of the surgical robot 11. For example, it can be configured that when the distance between the surgical robot 11 and the nearest obstacle to the surgical robot 11 is less than a preset safety distance, it is determined that the surgical robot 11 needs to adjust its motion state during motion. The surgical robot 11 or the control processing device can also be provided with prompting devices such as warning lamps or buzzers. When the control processing device determines that the movement state of the surgical robot 11 needs to be adjusted during the movement, it can output prompt information through the prompting device, so that the operator It is known that adjustment gestures need to be issued to control the surgical robot 11 to adjust the motion state to achieve the purpose of obstacle avoidance.

The operator can specifically control the movement direction of the surgical robot 11 when using adjustment gestures to control the surgical robot 11 to adjust the motion state, such as controlling the robot to turn left, right, backward, forward and so on. If the operator does not receive the prompt, the control processing device may control the surgical robot 11 to continue moving along the planned path until reaching the surgical operation position. In other examples, the ranging device may further measure the position coordinates of the surgical robot 11 in real time, and transmit the position coordinates of the surgical robot 11 to the control processing device. The control processing device can determine whether the position coordinates of the surgical robot 11 are on the planned path, and if it deviates from the planned path, it can also control the prompting device to issue a prompt.

In some examples, the distance measuring device collects the position information of the robot by means of ultrasonic distance measurement. The specific principles are as follows:

Ultrasonic distance measurement is realized by means of the ultrasonic pulse echo transit time method. Assuming that the time elapsed from the ultrasonic pulse from the sensor to the reception is t, and the propagation speed of the ultrasonic wave in the air is c, the distance from the sensor to the obstacle is The distance D can be obtained by formula (6):

D＝ct/2 (6)

It can be understood that if the IO (Trig (control terminal)) is used to trigger the ranging and give a high-level signal of at least 10us, the module will automatically transmit 8 square waves of 40khz, and automatically detect whether there is a signal return. If so, the ultrasonic output will be a high Level, the duration of the high level is the round-trip time of the ultrasonic wave (can be calculated by a timer), and the test distance = (high level time * speed of sound (340M/S))/2.

During the movement of the surgical robot 11, it can be configured such that when the distance between the surgical robot 11 detected by the ranging device and the nearest obstacle to the robot is less than a set safe distance, the prompting device will send out an alarm message. Subsequent operators can adjust the movement direction of the patient's operating end through gestures. For example, if it is less than 5cm, it is judged as the information prompt distance. When the distance between the patient operating end and the obstacle is less than 5cm, the buzzer in the prompt device will automatically sound and the red light will flash. At this time, it is necessary to adjust the movement direction through gestures.

In some examples, in step S14, when the operator judges whether the surgical robot 11 needs to adjust the motion state during the motion process, the robot preoperative navigation method may also include: projecting the planned path in the operating room according to the corresponding ratio between the initial position and between surgical locations.

Specifically, the control processing device forms a virtual map based on the real image in the operating room with a certain reduction ratio, and the planned path displayed on the virtual map is compared with the starting position and the surgical operation position of the actual control surgical robot 11 in the operating room. The real path when moving between should also be reduced by a certain ratio, so when projecting the planned path on the real scene, it is necessary to zoom in on a certain ratio according to the planned path on the virtual map, so that the planned path projected on the real scene can be Connects the starting position and the operating position in the operating room. In this way, the operator can observe in real time whether the surgical robot deviates from the planned path during the movement of the surgical robot 11 , and if it deviates, the operator can issue an adjustment gesture to control the surgical robot 11 to adjust the motion state.

In some examples, please refer to FIG. 14 , a mobile platform 111 and driving wheels 112 may be provided at the bottom of the surgical robot 11 . An example of the principle of the driver driving the robot to move is: given a point P between two wheels (the distance between each wheelbase point P is 1), the wheel radius r, the angle θ between the robot direction and the X-axis direction, and each wheel speed of

with

The overall velocity of the surgical robot 11 in the global frame of reference is predicted by the forward kinematics model.

It should be understood that although the various steps in the flowcharts of FIGS. 1 , 4 , 6 , 8 , 10 and 12 are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figures 1, 4, 6, 8, 10 and 12 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but may be performed at different times For execution, the execution order of these steps or stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

The present application also provides a robot preoperative navigation system. Please refer to FIG. 2 and FIG. 15 together. The robot preoperative navigation system includes a control processing device 31 and a human-computer interaction device 32 . The control processing device 31 is in communication connection with the human-computer interaction device 32 . The control processing device 31 is used to acquire the environment information in the operating room, and to generate a virtual map based on the environment information, in which a mark of the starting position of the surgical robot 11 and a mark of the operation position are formed. The human-computer interaction device 32 is used for displaying a virtual map, and for obtaining the first interaction information of the user. The control processing device 31 is further configured to generate a planned path of the surgical robot 11 in the virtual map based on the first interaction information, and the planned path is used for the surgical robot 11 to move from a corresponding initial position in the operating room to a surgical operation position.

In some examples, please also refer to FIG. 9 a , the human-computer interaction device 32 includes an AR device 20 , and the AR device 20 is used to overlay and display a virtual map in a real scene.

In some examples, the human-computer interaction device 32 is used to collect gesture images to obtain first interaction information. Specifically, it can be configured that the AR device 20 is provided with a binocular camera 21, and the binocular camera 21 is used to collect gesture images to obtain the first interaction information.

In some examples, please refer to FIG. 5 , the control processing device 31 includes a position adjustment unit 312 and a camera 311, the camera 311 is arranged on the position adjustment unit 312, the position adjustment unit 312 is used to adjust the shooting angle of the camera 311, and the camera 311 is used for Multiple pixel images taken at different positions in the operating room are collected to obtain environmental information.

In some examples, during the movement of the surgical robot 11 from the initial position to the surgical operation position in the operating room; the control processing device obtains the second interaction information based on the gesture information, and adjusts the movement of the surgical robot 11 based on the second interaction information. Exercise status, and/or give prompt information.

Optionally, the control processing device 31 is also communicatively connected with the surgical robot 11, and the control processing device 31 is also used to control the movement of the surgical robot 11 from the initial position to the surgical operation position in the operating room based on the planned path; if the surgical robot 11 moves If it is necessary to adjust the motion state during operation, the binocular camera 21 acquires gesture images, and the control processing device 31 obtains second interaction information based on the gesture information, and adjusts the motion state of the surgical robot 11 based on the second interaction information, and/or gives prompt information.

In some examples, the robot preoperative navigation system also includes a distance measuring device (not shown in the figure), the distance measuring device is arranged on the surgical robot 11, and the distance measuring device is used to measure the distance between the surgical robot 11 and the obstacle in real time; The control processing device 31 is also connected in communication with the ranging device, and the control processing device 31 is also used to judge whether the distance between the surgical robot 11 and the obstacle is less than a preset safety distance, and to determine whether the distance between the surgical robot 11 and the obstacle is less than a predetermined distance. When the distance is less than the preset safety distance, a prompt message will be output.

In some examples, the human-computer interaction device 32 is used to project the planned path between the starting position and the surgical operation position in the operating room according to a corresponding scale.

Further, the robotic preoperative navigation system can also perform any steps in the above robotic preoperative navigation method. For specific limitations on the robot preoperative navigation system, please refer to the above-mentioned limitations on the robot preoperative navigation method, which will not be repeated here. Each module in the above robot preoperative navigation system can be fully or partially realized by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules. Wherein, the computer equipment may include the AR equipment 20, the control processing device 31, the driver of the surgical robot 11, and the like.

The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

The present application also provides a computer device, including a memory and a processor; the processor stores a computer program that can run on the processor, and the processor implements any of the above embodiments when executing the computer program The steps of the method.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only represent several implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims

A method for preoperative navigation of a robot, comprising:

Acquiring environmental information in the operating room, and displaying a virtual map based on the environmental information; a mark of the starting position of the surgical robot and a mark of the surgical operation position are formed in the virtual map; and

Acquiring first interaction information, and generating a planned path of the surgical robot in the virtual map based on the first interaction information; the planned path is used for the surgical robot to correspondingly start in the operating room The position moves to the surgical operation position.
The robot preoperative navigation method according to claim 1, wherein the virtual map is superimposed and displayed in the real scene.
The robot preoperative navigation method according to claim 1, wherein the virtual map includes a grid mark, and the mark of the starting position and the mark of the operation operation position are both located at the intersection of the grid mark The planned path connects the mark of the starting position and the mark of the surgical operation position through the intersection marked as no obstacle in the grid mark.
The robot preoperative navigation method according to claim 1, wherein said acquiring environmental information in the operating room and displaying a virtual map based on said environmental information comprises:

obtaining the environmental information by using multiple pixel images taken at different positions in the operating room;

Constructing and displaying a virtual map with a grid mark according to the environmental information; wherein, some intersection points in the grid mark display the environmental information.
The robot preoperative navigation method according to claim 4, wherein the environmental information includes position coordinate information of obstacles in the operating room and the starting position; A virtual map of cell identification, including:

Based on the position coordinate information of the surgical operation position and the position coordinate information of the obstacle and the initial position, expand the coordinate information of a plurality of virtual points that do not overlap with each position coordinate information;

Using the coordinate information of each virtual point and each position coordinate information as the intersection point of the grid mark, generating and displaying the virtual map with the grid mark;

Alternatively, the shape of the grid in the virtual map with the grid mark includes a triangle.
The robot preoperative navigation method according to claim 1, wherein the generating the planned path of the surgical robot in the virtual map based on the first interaction information comprises:

Based on the first interaction information, determine at least one pose change information relative to the starting position, so as to obtain a planned path of the surgical robot;

The planned route is displayed on the virtual map.
The robot preoperative navigation method according to any one of claims 1 to 6, further comprising: during the movement of the surgical robot from the initial position to the surgical operation position,

Acquire second interaction information, and adjust the motion state of the surgical robot based on the second interaction information, and/or give prompt information.
The robot preoperative navigation method according to claim 2, further comprising:

The planned path is projected in the operating room between the starting position and the surgical operation position in a corresponding scale.
The robot preoperative navigation method according to claim 7, wherein the need to adjust the motion state during the movement of the surgical robot includes: the distance between the surgical robot and the obstacle is less than a preset safety distance and/or the The trajectory of the surgical robot deviates from the planned path.
The robot preoperative navigation method according to claim 1, further comprising: controlling the movement of the surgical robot and/or controlling the positioning of the mechanical arm of the surgical robot based on the planned path.
A robot preoperative navigation system, comprising: a control processing device and a human-computer interaction device, the control processing device and the human-computer interaction device are connected in communication;

The human-computer interaction device is used to display a virtual map, and to obtain the user's first interaction information;

The control processing device is used to execute the robot preoperative navigation method according to any one of claims 1-10.
The robot preoperative navigation system according to claim 11, wherein the human-computer interaction device includes an AR device, and the AR device is used to overlay and display the virtual map in a real scene.
The robot preoperative navigation system according to claim 11, wherein the control processing device includes a position adjustment unit and a camera, the camera is arranged on the position adjustment unit, and the position adjustment unit is used to adjust the position of the camera The camera is used to collect multiple pixel images taken at different positions in the operating room to obtain the environmental information.
The robot preoperative navigation system according to claim 11, wherein the human-computer interaction device projects the planned path in the operating room between the initial position and the surgical operation position according to a corresponding proportion .
The robot preoperative navigation system according to any one of claims 11 to 14, wherein, during the movement of the surgical robot from the initial position to the surgical operation position in the operating room;

The control processing device obtains the second interaction information based on the gesture information, and adjusts the motion state of the surgical robot and/or gives prompt information based on the second interaction information.
The robot preoperative navigation system according to claim 11, wherein the human-computer interaction device acquires the first interaction information by collecting gesture images.
A storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1-10 are implemented.
A computer device, comprising a memory and a processor; the processor stores a computer program that can run on the processor, and when the processor executes the computer program, any one of claims 1 to 10 is realized The steps of the method.