TWI825499B - Surgical robotic arm control system and control method thereof - Google Patents

Abstract

A surgical robotic arm control system and a control method thereof are provided. The surgical robotic arm control system includes a surgical robotic arm, an image capture unit and a processor. The surgical robot arm has multiple joint axes. The image capturing unit obtains a first image. The processor executes a spatial environment recognition module to generate a first environment information image, a first direction information image, and a first depth information image based on the first image. The processor executes a spatial environment image processing module to calculate path information based on the first environment information image, the first direction information image, and the first depth information image. The processor executes the robot arm motion feedback module to operate the surgical robot arm to move according to the path information.

Description

Surgical robotic arm control system and control method thereof

The present invention relates to an automatic control technology, and in particular to a surgical robotic arm control system and a control method thereof.

With the evolution of medical equipment, automatically controllable related medical equipment that can help supplement the surgical efficiency of medical personnel is currently one of the important development directions in this field. In particular, the surgical robotic arm used to assist or cooperate with medical personnel (operators) in performing related surgical work during the operation is even more important. However, in the existing surgical robotic arm design, in order for the surgical robotic arm to achieve automatic control functions, the surgical robotic arm must be equipped with multiple sensors and the user must perform cumbersome manual correction operations during each operation. , so that the surgical robotic arm can avoid obstacles in the path during movement and achieve accurate automatic movement and automatic operation results.

The invention provides a surgical robotic arm control system and a control method thereof, which can effectively control the surgical robotic arm to move automatically.

The surgical robot arm control system of the present invention includes a surgical robot arm, an image capture unit and a processor. Surgical robotic arms have multiple joint axes. The image capturing unit acquires the first image. The first image includes an image of the end of the robotic arm of the surgical robotic arm. The processor is coupled to the surgical robotic arm and the image capture unit. The processor executes the spatial environment recognition module to generate a first environment information image, a first direction information image and a first depth information image based on the first image. The processor executes the spatial environment image processing module to calculate the path information based on the first environment information image, the first direction information image and the first depth information image. The processor executes the robotic arm motion feedback module to operate the surgical robotic arm movement based on the path information.

The method for controlling a surgical robotic arm of the present invention includes the following steps: obtaining a first image through an image acquisition unit, where the first image includes an image of the end of the robotic arm of the surgical robotic arm; executing a spatial environment recognition module through a processor to determine The first image generates a first environment information image, a first direction information image and a first depth information image; the processor executes the spatial environment image processing module to generate a first environment information image, a first direction information image and a first depth information image according to the first environment information image, the first direction information image and the first depth information image. The depth information image estimates the path information; and the processor executes the robotic arm motion feedback module to operate the surgical robotic arm movement based on the path information.

Based on the above, the surgical robotic arm control system and its control method of the present invention can automatically control the movement of the surgical robotic arm through brain vision imaging technology, and can automatically avoid obstacles in the current environment.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

In order to make the content of the present invention easier to understand, the following embodiments are provided as examples according to which the present disclosure can be implemented. In addition, wherever possible, elements/components/steps with the same reference numbers in the drawings and embodiments represent the same or similar parts.

Figure 1 is a schematic circuit diagram of a surgical robotic arm control system according to an embodiment of the present invention. Referring to FIG. 1 , the surgical robot arm control system 100 includes a processor 110 , a storage unit 120 , an image capture unit 130 and a surgical robot arm 140 . The storage unit 120 stores a space environment recognition module 121, a space environment image processing module 122, and a robot arm motion feedback module 123. The processor 110 is coupled to the storage unit 120, the image capture unit 130 and the surgical robotic arm 140. Surgical robotic arm 140 has multiple joint axes. In this embodiment, the image capture unit 130 can obtain image data and provide it to the processor 110 . The processor 110 can access the storage unit 120 to execute the spatial environment recognition module 121, the spatial environment image processing module 122, and the robotic arm motion feedback module 123. In this embodiment, the processor 110 can input relevant image data to the spatial environment recognition module 121 and the spatial environment image processing module 122 to generate path information, and the processor 110 can operate the surgical robotic arm 140 according to the path information. Move.

In this embodiment, the surgical robot arm control system 100 can be integrated with the mechanism of the surgical platform. The image capturing unit 130 can be disposed on the upper side of the surgical platform (directly above or offset at an angle) to take pictures toward the surgical platform and the surgical robotic arm 140 . Moreover, the surgical robot arm 140 may be disposed on one side of the surgical platform. In this embodiment, the surgical robot arm control system 100 can control the surgical robot arm 140 to move from one side of the surgical platform to the other end of the surgical platform, and the surgical robot arm 140 and its end can automatically avoid obstacles on the movement path. Obstacles. Therefore, the operator can quickly master the surgical assisting function of the surgical robotic arm 140 at the other end of the surgical platform.

In this embodiment, the processor 110 may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor) or digital signal processor (Digital Signal Processor). Processor (DSP), Image Processing Unit (IPU), Graphics Processing Unit (GPU), programmable controller, Application Specific Integrated Circuits (ASIC), programmable Logic device (Programmable Logic Device, PLD), other similar processing devices, or a combination of these devices.

In this embodiment, the storage unit 120 can be a memory, such as a dynamic random access memory (Dynamic Random Access Memory, DRAM), a flash memory (Lash memory) or a non-volatile random access memory. (Non-Volatile Random Access Memory, NVRAM), which is not limited by the present invention. The storage unit 120 can store the spatial environment recognition module 121, the spatial environment image processing module 122, the robotic arm action feedback module 123, and the related algorithms of each module mentioned in various embodiments of the present invention, and can also store image data. , robotic arm control instructions, robotic arm control software, computing software and the like related algorithms, programs and data used to implement the surgical robotic arm control function of the present invention. In this embodiment, the spatial environment recognition module 121 and the spatial environment image processing module 122 can respectively be neural network modules that implement corresponding functions.

In this embodiment, the surgical robotic arm 140 may be a robotic arm with six degrees of freedom tracking (6DOF), and the processor 110 may execute a machine learning module that applies a Markov decision process to control the surgery. Robotic arm 140. In this embodiment, the image capture unit 130 can be, for example, a depth camera, and can be used to capture the surgical field to obtain field images and depth information. In one embodiment, the storage unit 120 can also store the panoramic environment field positioning module. The processor 110 can execute the panoramic environment field positioning module to perform camera calibration calculations, and can implement the coordinate system matching function between the image capture unit 130 and the surgical robotic arm 140 . In this embodiment, the image capturing unit 130 can obtain positioning images and reference depth information in advance, where the positioning images include positioning objects. The processor 110 can analyze the positioning coordinate information and the reference depth information of the positioning object in the positioning image through the panoramic environment field positioning module, so that the camera coordinate system of the image capture unit 130 (depth camera) is consistent with the mechanism of the surgical robotic arm 140 Arm coordinate system matching.

Specifically, the user can, for example, use a positioning plate with a checkerboard pattern as a positioning object and place it on the surgical platform so that the image capture unit 130 can capture multiple positioning images, wherein each of these positioning images can be Pattern including checkerboard image. The number of positioning images may be, for example, five. Then, the processor 110 can execute the panoramic environment field positioning module to analyze the positioning coordinate information (multiple spatial coordinates) and reference depth information of respective positioning objects in the multiple positioning images through the panoramic environment field positioning module, so as to The camera coordinate system (spatial coordinate system) of the image capture unit 130 is matched with the robot arm coordinate system (spatial coordinate system) of the surgical robot arm 140 . The processor 110 can match the camera coordinate system of the image capture unit 130 and the robotic arm coordinate system of the surgical robotic arm 140 based on the fixed position relationship, positioning coordinate information, and reference depth information.

FIG. 2 is an operational schematic diagram of a surgical robotic arm control system according to an embodiment of the present invention. Figure 3 is a flow chart of a method for controlling a surgical robotic arm according to an embodiment of the present invention. FIG. 4 is a schematic diagram of image processing and image analysis according to an embodiment of the present invention. Referring to FIGS. 1 to 4 , the image capturing unit 130 may, for example, take pictures toward a surgical platform, on which a surgical object 200 may be placed, for example. In this embodiment, the surgical robot arm 140 may be located on one side of the surgical object 200 as shown in FIG. 2 , and the processor 110 may control the surgical robot arm 140 to move to the other side of the surgical area 201 of the surgical object 200 , and may Obstacles on the movement path in the surgical area 201 are automatically avoided, where the obstacles may include, for example, surgical instruments 202 to 204 placed on the surgical object 200 .

In this embodiment, the surgical robot arm control system 100 can perform the following steps S310 to S340. In step S310 , the surgical robot arm control system 100 can obtain the first image 401 (current frame) through the image capture unit 130 , where the first image 401 includes an image of the end of the robot arm of the surgical robot arm 140 . In this embodiment, the storage unit 120 may also store the target area confirmation module, and the surgical robot arm control system 100 may further include an input unit. The input unit may be, for example, a mouse, a touch screen, a user interface or a system setting module, and may provide target coordinates to the processor 110 . In this regard, the processor 110 may execute the target area confirmation module to define the target area in the first image 401 according to the target coordinates. In this regard, the target area is a spatial area (virtual cube), and may be, for example, on the other side of the surgical object in the first image 401 .

In step S320, the surgical robot arm control system 100 can execute the spatial environment recognition module 121 through the processor 110 to generate the first environment information image 411, the first direction information image 412 and the first depth information image according to the first image 401. 413. In step S330, the surgical robot arm control system 100 can use the processor 110 to execute the spatial environment image processing module 122 to calculate the path information based on the first environment information image 411, the first direction information image 412 and the first depth information image 413. . In this embodiment, the spatial environment image processing module 122 can respectively capture the first environmental information image 411, the first direction information image 412 and the first depth information image 413 according to the end area of the robotic arm of the surgical robotic arm 140. The second environment information image 421, the second depth information image 422 and the second direction information image 423 (only the image part of the end of the robotic arm in the image is captured for subsequent calculation and analysis). In this regard, since the second environment information image 421, the second depth information image 422 and the second direction information image 423 are parts of the first environment information image 411, the first direction information image 412 and the first depth information image 413 respectively, therefore The surgical robot arm control system 100 of the present invention can perform rapid image calculation and analysis on the key areas of each frame of images, thereby effectively saving computing resources and quickly calculating to move the surgical robot arm 140 to the target coordinates.

For example, the first environment information image 411, the first direction information image 412, and the first depth information image 413 may each have an image resolution of 224×224 pixels, and the second environment information image 421, the second depth information image 422 and the second direction information image 423 may each have an image resolution of 54×54 pixels. Before the spatial environment image processing module 122 inputs the second environment information image 421, the second depth information image 422, and the second direction information image 423 to the fully convolutional network model 122, the spatial environment image processing module 122 may first separately The second environment information image 421, the second depth information image 422 and the second direction information image 423 perform image amplification, where the image amplification may be performed, for example, through a bilinear interpolation algorithm. The enlarged second environment information image 431, the enlarged second depth information image 432, and the enlarged second direction information image 433 may each have an image resolution of 224×224 pixels. Next. The spatial environment image processing module 122 can input the enlarged second environment information image 431, the enlarged second depth information image 432, and the enlarged second direction information image 433 to the fully convolutional network model 122, so that The fully convolutional network model 122 outputs feature images 451.

The fully convolutional network model 122 may include a dense neural network module 122-1 (the upper part of the operation model) and a feature reduction module 122-2 (the lower part of the operation model). The dense neural network module 122-1 can first generate multiple feature value data 441-1~441-N, 442-1~442-N, and 443-1~443-N of the training results. The feature value data 441-1~441-N may be the training results of the enlarged second environment information image 431. The feature value data 442-1~442-N may be the training results of the enlarged second depth information image 432. The feature value data 443-1~443-N may be the training results of the enlarged second direction information image 433. The fully convolutional network model 122 can then input the feature value data 441-1~441-N, 442-1~442-N, 443-1~443-N to the feature restoration module 122-2 to restore the features The module 122-2 can reorganize the feature value data 441-1~441-N, 442-1~442-N, 443-1~443-N to output the feature image 451. In this embodiment, the spatial environment image processing module 122 can analyze the characteristic image 451 to derive path information. The feature image 451 may, for example, have weight distribution information (movable weight or obstacle weight) corresponding to the position of each point in space or a moving plane. Furthermore, the processor 110 can, for example, calculate information or parameters such as the direction in which the surgical robot arm 140 can move and the distance it can move in the current frame based on the characteristic image 451 .

In step S340, the surgical robot arm control system 100 can use the processor 110 to execute the robot arm action feedback module 123 to operate the surgical robot arm 140 to move to the target area according to the path information. In this embodiment, the image capture unit 130 can continuously acquire multiple first images of multiple frames, so that the processor 110 iteratively executes the spatial environment recognition module 121 and the spatial environment image processing module based on these first images. 122 and the robotic arm motion feedback module 123 to operate the surgical robotic arm 140 multiple times until the processor 110 determines that the end of the robotic arm of the surgical robotic arm 140 reaches the target coordinates. In this regard, when the processor 110 can determine that the robotic arm end area of the surgical robotic arm 140 overlaps with the target area (two virtual cubes overlap), the processor 110 can determine that the robotic arm end of the surgical robotic arm 140 has reached the target coordinates. The end area of the robot arm may be a cubic area simulated by the processor 110 based on the center point of the spatial position of the end of the robot arm and extending outward from the center (the center point of the area is the center point of the end of the robot arm). Therefore, the surgical robot arm 140 can automatically avoid the surgical instruments 202 to 204 on the movement path to automatically move to the other side of the surgical object 200 .

The following embodiments of FIGS. 5 to 7 will respectively describe in detail the generation method of the second environment information image 421, the second depth information image 422, and the second direction information image 423.

FIG. 5 is a schematic diagram of generating a second environmental information image according to an embodiment of the present invention. Referring to FIG. 1 and FIG. 5 , the image capture unit 130 may, for example, capture the surgical field 501 as shown in FIG. 5 to obtain the first environmental information image 502 (ie, the first image). The processor 110 may define the position of the robotic arm end corresponding to the surgical robotic arm in the first environmental information image 502 to determine the range of the robotic arm end region 511 (preset analysis range). In this regard, the horizontal range of the robot arm end area 511 may correspond to the range 512 in the first environmental information image 502 . Then, the processor 110 can crop the first environmental information image 502 according to the range 512 to generate a second environmental information image 421 (RGB image).

FIG. 6 is a schematic diagram of generating a second depth information image according to an embodiment of the present invention. Referring to FIG. 1 and FIG. 6 , the image capture unit 130 may, for example, capture the surgical field 601 as shown in FIG. 6 to obtain a first depth information image with depth information (ie, a first image with depth information). . The processor 110 may define the position of the robotic arm end corresponding to the surgical robotic arm in the first depth information image to determine a reference plane based on the extension axis 611 of the robotic arm end 141 . The first depth information image may include a plurality of first depth plane images 602_1˜602_N corresponding to different depths, where N is a positive integer. The different depths may, for example, refer to the reference plane and 5 vertical depths above and below the reference plane (for example -5, -4, -3, -2, -1, 0, +1, +2, +3, +4, +5) plane, but the present invention does not limit the number of samples of depth plane images. In this regard, the horizontal range of the robot end area of the robot end 141 (like the robot end area 511 of FIG. 5 ) may correspond to the range 512 of the same position in the plurality of first depth plane images 602_1˜602_N. Then, the processor 110 can convert these first depth plane images 602_1 ~ 602_N into a plurality of binary images 603_1 ~ 603_N (for example, if there are obstacles, they are represented by the value "0" (pure black), and if there are no obstacles, they are represented by the value "0" (pure black)). "1" represents (pure white)), and a plurality of second depth plane images 422_1 corresponding to different depths in the second depth information image are obtained from these binary images 603_1~603_N according to the end area of the robotic arm of the surgical robotic arm 140 ~422_N. In this regard, the surgical robot arm control system 100 can obtain obstacle distribution information (such as distribution information of other surgical instruments) at different depth planes based on the second depth plane images 422_1 ~ 422_N, so as to effectively calculate that the surgical robot arm 140 will not The path of movement that hits an obstacle.

FIG. 7 is a schematic diagram of generating a first direction information image according to an embodiment of the present invention. Referring to FIG. 1 and FIG. 7 , the image capturing unit 130 may, for example, capture the surgical field 701 shown in FIG. 7 to capture the first environmental information image 702 (ie, the first image). The processor 110 may define the end of the robotic arm corresponding to the surgical robotic arm in the first environment information image 702 to determine the end point of the end of the robotic arm as the end point P1 of the robotic arm of the current frame. The processor 110 may define the robotic arm end of the surgical robotic arm in the first environmental information image 702 to determine the range of the robotic arm end area (preset analysis range). Furthermore, the processor 110 can obtain the target coordinates according to the target selection signal provided by the input unit (for example, the user selects the target position) to determine the target point P2. The processor 110 may determine the radioactive gradient color parameters along the path from the robot arm end point P1 to the target point P2 in the first environment information image 702 and at different distances toward the target point P2 to generate the first direction information image 703 . It is worth noting that the color change direction of the first direction information image 703 is parallel to the direction from the robot end coordinate of the robot end point P1 to the target coordinate of the target point P2 on the first direction information image 703 . The processor 110 may crop the first direction information image 703 according to the range 512 to generate the second direction information image 423.

To sum up, the surgical robot arm control system and its control method of the present invention can realize the use of computer vision imaging technology through the image capture unit to automatically control the movement of the surgical robot arm and approach the target object, and can centralize computing resources. By performing calculations and analyzing key areas in the sensing image provided by the image capture unit, rapid and accurate surgical robotic arm control can be achieved. Therefore, the surgical robot arm control system and its control method of the present invention can effectively enable the surgical robot arm to automatically move to, for example, a position adjacent to the operator's hand or the surgical object, so that the operator can perform surgery quickly and efficiently. Robotic arm to achieve its surgical assistance function.

Although the present invention has been disclosed above through embodiments, they are not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some modifications and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the appended patent application scope.

100:Surgical robotic arm control system 110: Processor 120:Storage unit 121: Space environment recognition module 122: Space environment image processing module 122-1: Dense Neural Network Module 122-2: Feature restoration module 123: Robotic arm action feedback module 130:Image capture unit 140:Surgical robotic arm 200:Surgery object 201:Surgery area 202~204: Surgical instruments 401:First Image 411, 502, 702: First environmental information image 412, 703: First direction information image 413: The first depth information image 421: Second environmental information image 422: Second depth information image 422_1~422_N: Second depth plane image 423: Second direction information image 431: Enlarged second environment information image 432: Magnified second depth information image 433: Enlarged second direction information image 441-1~441-N, 442-1~442-N, 443-1~443-N: Characteristic value data 451:Characteristic image 501, 601, 701: Surgery field 511: Robotic arm end area 512: Range 602_1~602_N: First depth plane image 603_1~603_N: Binarized image 611:Extended shaft P1: End point of robot arm P2: target point S310~S340: steps

Figure 1 is a schematic circuit diagram of a surgical robotic arm control system according to an embodiment of the present invention. FIG. 2 is an operational schematic diagram of a surgical robotic arm control system according to an embodiment of the present invention. Figure 3 is a flow chart of a method for controlling a surgical robotic arm according to an embodiment of the present invention. FIG. 4 is a schematic diagram of image processing and image analysis according to an embodiment of the present invention. FIG. 5 is a schematic diagram of generating a second environmental information image according to an embodiment of the present invention. FIG. 6 is a schematic diagram of generating a second depth information image according to an embodiment of the present invention. FIG. 7 is a schematic diagram of generating a second direction information image according to an embodiment of the present invention.

100:Surgical robotic arm control system

110: Processor

120:Storage unit

121: Space environment recognition module

122: Space environment image processing module

123: Robotic arm action feedback module

130:Image capture unit

140:Surgical robotic arm

Claims (20)

A surgical robot arm control system, including: a surgical robot arm with multiple joint axes; an image capture unit used to obtain a first image, wherein the first image includes an end-of-arm image of the surgical robot arm ; and a processor coupled to the surgical robotic arm and the image capture unit, wherein the processor executes a spatial environment recognition module to generate a first environmental information image and a first direction information based on the first image image and a first depth information image, and the processor executes a spatial environment image processing module to calculate a path information based on the first environment information image, the first direction information image and the first depth information image, wherein The processor executes a robotic arm motion feedback module to operate the surgical robotic arm to move according to the path information. The surgical robot arm control system as claimed in claim 1, wherein the image capture unit is a depth camera, and the image capture unit pre-obtains a positioning image and a reference depth information, wherein the positioning image includes a positioning object, The processor executes a panoramic environment field positioning module to analyze a positioning coordinate information and the reference depth information of the positioning object in the positioning image through the panoramic environment field positioning module, so as to make the depth camera A camera coordinate system matches a robotic arm coordinate system of the surgical robotic arm. The surgical robot arm control system according to claim 1, wherein the processor executes a target area confirmation module to define the first image according to a target coordinate. A target area in the image, and the robotic arm motion feedback module operates the surgical robotic arm to move to the target area. The surgical robot arm control system according to claim 3, wherein the spatial environment image processing module obtains the first environment information image, the first direction information image and the third according to a robot arm end area of the surgical robot arm. A depth information image captures a second environment information image, a second direction information image and a second depth information image respectively, and combines the second environment information image, the second direction information image and the second depth information image Input to a fully convolutional network model, so that the fully convolutional network model outputs a feature image, and the processor generates the path information based on the feature image, wherein the second environment information image, the second direction information The image and the second depth information image are respectively part of the first environment information image, the first direction information image and the first depth information image. The surgical robot arm control system as claimed in claim 4, wherein before the processor inputs the second environment information image, the second direction information image and the second depth information image to the fully convolutional network model, The processor first magnifies the second environment information image, the second direction information image and the second depth information image respectively. The surgical robot arm control system according to claim 4, wherein the first depth information image includes a plurality of first depth plane images corresponding to different depths, and the processor converts the first depth plane images into a plurality of two-dimensional images. value image, and the root A plurality of second depth plane images corresponding to different depths in the second depth information image are obtained from the binary images according to the end area of the robotic arm of the surgical robotic arm. The surgical robot arm control system as claimed in claim 4, wherein when the processor determines that the robot arm end area of the surgical robot arm overlaps the target area, the processor determines that the robot arm end area of the surgical robot arm has reached The target coordinates. The surgical robot arm control system according to claim 7, wherein the image capture unit continuously acquires a plurality of first images, so that the processor iteratively executes the spatial environment recognition module, the spatial environment recognition module and the spatial environment recognition module according to the first images. The environmental image processing module and the robotic arm motion feedback module operate the surgical robotic arm multiple times until the processor determines that the end of the robotic arm of the surgical robotic arm reaches the target coordinate. The surgical robot arm control system of claim 4, wherein a color change direction of the second direction information image is parallel to a direction from a robot arm end coordinate to the target coordinate on the second direction information image. The surgical robot arm control system according to claim 3, further comprising: an input unit coupled to the processor and providing a target selection signal to the processor, so that the processor generates the target according to the target selection signal. coordinates. A method for controlling a surgical robotic arm, including: obtaining a first image through an image acquisition unit, wherein the first image includes a robotic arm end image of a surgical robotic arm; executing a spatial environment recognition model through a processor Group to generate a first environment information image, a first direction information image and a first depth information image based on the first image; A spatial environment image processing module is executed by the processor to calculate a path information based on the first environment information image, the first direction information image and the first depth information image; and a machine is executed by the processor The arm motion feedback module is used to operate the surgical robot arm to move according to the path information. The surgical robot arm control method as described in claim 11, wherein the image capture unit is a depth camera, and the image capture unit obtains a positioning image and a reference depth information in advance, wherein the positioning image includes a positioning object, The surgical robot arm control method includes the following steps: using the processor to execute a panoramic environment field positioning module to analyze a positioning coordinate information of the positioning object in the positioning image through the panoramic environment field positioning module and the reference depth information to match a camera coordinate system of the depth camera with a robotic arm coordinate system of the surgical robotic arm. The surgical robot arm control method according to claim 11, further comprising: executing a target area confirmation module by the processor to define a target area in the first image according to a target coordinate, and the robot arm The motion feedback module operates the surgical robotic arm to move to the target area. The surgical robot arm control method according to claim 13, wherein the spatial environment image processing module is executed by the processor to control the surgical robot arm according to the first environment information image, the first direction information image and the first depth information image. The steps to derive this path information include: The spatial environment image processing module acquires a second environment information respectively from the first environment information image, the first direction information image and the first depth information image according to a robot arm end area of the surgical robot arm. image, a second direction information image and a second depth information image; and input the second environment information image, the second direction information image and the second depth information image to a A fully convolutional network model, so that the fully convolutional network model outputs a characteristic image, and the processor generates the path information based on the characteristic image, wherein the second environment information image and the second direction information image And the second depth information image is a part of the first environment information image, the first direction information image and the first depth information image respectively. The surgical robot arm control method as claimed in claim 14, wherein before the processor inputs the second environment information image, the second direction information image and the second depth information image to the fully convolutional network model, The processor first magnifies the second environment information image, the second direction information image and the second depth information image respectively. The surgical robot arm control method according to claim 14, wherein the first depth information image includes a plurality of first depth plane images corresponding to different depths, and the step of generating the second depth information image includes: by the processing The device converts the first depth plane images into a plurality of binary images, and obtains the second depth information images corresponding to different depths from the binary images according to the end area of the robotic arm of the surgical robot arm. multiple second depths Flat image. The surgical robot arm control method as described in claim 14, further comprising: when the processor determines that the robot end area of the surgical robot arm overlaps with the target area, the processor determines the end area of the surgical robot arm. The end of the robotic arm reaches the target coordinates. The surgical robot arm control method as described in claim 17, further comprising: continuously acquiring a plurality of first images through the image acquisition unit, so that the processor iteratively executes the spatial environment recognition model based on the first images. The group, the spatial environment image processing module and the robotic arm motion feedback module operate the surgical robotic arm to move multiple times until the processor determines that the end of the robotic arm of the surgical robotic arm reaches the target coordinates. The surgical robot arm control method as claimed in claim 13, wherein a color change direction of the second direction information image is parallel to a direction from a robot arm end coordinate to the target coordinate on the second direction information image. The surgical robot arm control method according to claim 13, further comprising: providing a target selection signal to the processor through an input unit, so that the processor generates the target coordinates according to the target selection signal.

Images