TWI801775B - Automatic control method of mechnical arm and automatic control system - Google Patents

Abstract

An automatic control method of a mechanical arm and automatic control system are provided. The automatic control method includes the following steps: obtaining a color image and depth information corresponding to the color image through a depth camera; performing image space cutting processing and image rotation processing based on the color image and the depth information to generate multiple depth images; inputting the depth image to an environmental image recognition module, so that the environmental image recognition module outputs a displacement coordinate parameter; and outputting the displacement coordinate parameter to a robot control module, so that the robot control module controls a movement of the mechanical arm according to the displacement coordinate parameter.

Description

Automatic control method and automatic control system of mechanical arm

The present invention relates to a control method and system, and in particular to an automatic control method and an automatic control system of a mechanical arm.

With the evolution of medical equipment, related medical equipment that can be automatically controlled, which can help to increase the surgical efficiency and accuracy of medical personnel, is currently one of the important development directions in this field. In particular, the mechanical arm used to assist or cooperate with medical personnel (performers) in performing relevant surgical work during the operation is more important. However, in the existing design of the robotic arm, in order to realize the automatic control function of the robotic arm, the robotic arm must be equipped with multiple sensors and the user must perform cumbersome manual calibration operations during each operation. Make the robot arm avoid obstacles in the path during the movement process, and realize accurate automatic movement and automatic operation results. In view of this, the following will propose a new type of automatic control system design.

The invention provides an automatic control method and an automatic control system of a mechanical arm, which can operate the mechanical arm to move in space and effectively avoid obstacles.

The automatic control method of the robotic arm of the present invention includes the following steps: obtaining a color image and depth information corresponding to the color image through a depth camera; performing image space cutting and image rotation processing according to the color image and depth information to generate multiple depth images ; Input the plurality of depth images to the environment image recognition module, so that the environment image recognition module outputs displacement coordinate parameters; and output the displacement coordinate parameters to the robot arm control module, so that the robot arm control module according to the displacement coordinate Parameters to control the movement of the robot arm.

The automatic control system of the mechanical arm of the present invention includes a depth camera and a processor. The depth camera is used to obtain color images and depth information corresponding to the color images. The processor is coupled to the robotic arm and the depth camera. The processor is used for image space cutting and image rotation processing according to the color image and depth information, so as to generate multiple depth images. The processor inputs the plurality of depth images to the environment image recognition module, so that the environment image recognition module outputs displacement coordinate parameters. The processor outputs the displacement coordinate parameter to the robot arm control module, so that the robot arm control module controls the movement of the robot arm according to the displacement coordinate parameter.

Based on the above, the automatic control method and automatic control system of the robotic arm of the present invention can realize the function of automatically judging obstacles in the current environment through visual training, and can effectively operate the robotic arm to move in the current environment.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the following special citations Embodiments, together with the accompanying drawings, are described in detail as follows.

100: Automatic control system

110: Processor

120: memory

121:Environmental image recognition module

122: Robotic arm control module

130: Depth camera

140: Mechanical arm

141: fixture

150: display device

200: Surgical object

210: Surgical position

410: Control module

411: Image processing module

412: Track record module

413: Feedback Module

414: Output module

420: Training module

600: color image

601: range

610:Color image

620: Environment image

630: target position

700_1~700_N: Depth image

S310~S340, S510~S570, S810, S820: steps

800_1~800_N: effective safe space image

900_1~900_N: moving track

FIG. 1 is a schematic block diagram of an automatic control system according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of the operation of the automatic control system according to an embodiment of the present invention.

FIG. 3 is a flowchart of an automatic control method according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of multiple modules of a neural network operation according to an embodiment of the present invention.

Fig. 5 is a flowchart of an automatic control method according to another embodiment of the present invention.

FIG. 6 is a schematic diagram of a color image according to an embodiment of the invention.

FIG. 7 is a schematic diagram of a plurality of depth images according to an embodiment of the invention.

FIG. 8 is a flow chart of training a module of a neural network operation according to an embodiment of the present invention.

FIG. 9 is a schematic diagram of a plurality of effective safe space images according to an embodiment of the present invention.

In order to make the content of the present invention more comprehensible, the following specific examples are given as examples in which the present invention can indeed be implemented. In addition, wherever possible, in Elements/components/steps using the same reference numerals in the drawings and embodiments represent the same or similar components.

FIG. 1 is a schematic block diagram of an automatic control system according to an embodiment of the present invention. Referring to FIG. 1 , the automatic control system 100 includes a processor 110 , a memory 120 and a depth camera 130 . The processor 110 is coupled to the memory 120 , the depth camera 130 and the robot arm 140 . The robotic arm 140 can be a multi-axis robotic arm (eg, six-axis). In this embodiment, the memory 120 can store the environment image recognition module 121 and the robot arm control module 122 . The processor 110 can access the memory 120 and execute the environment image recognition module 121 and the robot arm control module 122 to control the movement and related operations of the robot arm 140 . In this embodiment, the processor 110 and the memory 120 can be integrated into a host computer, for example, and communicate with the depth camera 130 and the robotic arm 140 in a wired or wireless manner. However, in one embodiment, the processor 110 and the memory 120 can also be integrated into the cloud server system, but the present invention is not limited thereto.

In this embodiment, the processor 110 can first obtain the color image corresponding to the target position and the robot arm and the depth information corresponding to the color image through the depth camera 130, and then execute according to the color image and the depth information corresponding to the color image. The environment image recognition module 121 recognizes the environment of the target location through computer vision image processing. The processor 110 can execute the robot arm control module 122 according to the displacement or path parameters corresponding to the environment recognition result output by the environment image recognition module 121 , so that the robot arm control module 122 can generate corresponding control signals to the robot arm 140 . In this embodiment, the robotic arm control module 122 may include The input interface of the control robot arm 140 (such as socket or API, etc.), and the robot arm control module 122 can perform the calculation of forward and inverse kinematics of the robot arm 140 . Therefore, the robot arm control module 122 can control the robot arm 140 to automatically move to the target position in space, and can effectively avoid obstacles in the environment.

In this embodiment, the processor 110 may include a central processing unit (Central processing unit; CPU), a programmable general-purpose or special-purpose microprocessor (Microprocessor), a digital signal processor (Digital signal processor; DSP) , programmable controller, application specific integrated circuit (Application specific integrated circuit; ASIC), graphics processor (Graphics processing unit, GPU) or other similar components or a combination of the above components and can be used to implement the relevant functional circuits of the present invention.

In this embodiment, the memory 120 may include, for example, random access memory (Random-Access Memory; RAM), read-only memory (Read-Only Memory; ROM), optical disc (Optical disc), magnetic disk (Magnetic disk) ), Hard drive, Solid-state drive, Flash drive, Security digital (SD) card, Memory stick, Compact flash ( Compact flash; CF card or any type of storage device. In this embodiment, the memory 120 can be used to store related modules, related image data, and related parameters mentioned in various embodiments of the present invention, so that the processor 110 can access the memory 120 to execute related data. processing and computing.

FIG. 2 is a schematic diagram of the operation of the automatic control system according to an embodiment of the present invention. Referring to Fig. 1 and Fig. 2, the automatic control system 100 of Fig. 1 can be applied to As shown in FIG. 2 , the scenario of a medical operation, and the robot arm 140 may be a surgical robot arm. As shown in FIG. 2 , the processor 110 of the automatic control system 100 may be further coupled to a display device 150 . Specifically, the depth camera 130 can shoot towards the operation site 210 of the operation object 200 to continuously acquire a plurality of color images of the operation position 210 and a plurality of depth information corresponding to the plurality of color images. The depth camera 130 can provide the plurality of color images and a plurality of depth information corresponding to the plurality of color images to the processor 110 . The processor 110 can display relevant real-time information on the display device 150 according to the shooting result of the depth camera 130 for medical personnel to judge or monitor, and the present invention does not limit the display content of the display device 150 . In this embodiment, the processor 110 can perform image processing on each color image and the corresponding depth information to generate corresponding control signals to the robotic arm 140 so that the robotic arm 140 can automatically move towards the target in the surgical site 210 Location.

During the continuous shooting process of the depth camera 130 , the processor 110 also performs continuous visual image processing to continuously output control signals to the robotic arm 140 . In other words, the processor 110 can correspondingly control the movement of the robot arm 140 corresponding to the current environmental situation or environmental changes. For example, the processor 110 can control the gripper 141 of the manipulator arm 140 to move towards a specific medical device located in the surgical location 210, and during the movement of the robotic arm 140, the robotic arm 140 can automatically avoid obstacles in the surgical environment (including the body part of the surgical object 200), so that the gripper 141 of the robotic arm 140 can smoothly grip the specific medical device at the target position. Therefore, the automatic control system 100 of this embodiment can control the robotic arm 140 to effectively assist or cooperate with medical personnel to perform related operations.

FIG. 3 is a flowchart of an automatic control method according to an embodiment of the present invention. Referring to FIG. 1 to FIG. 3 , the automatic control system 100 can control the robot arm 140 to move automatically by executing steps S310 - S340 . In step S310 , the processor 110 can obtain a color image and depth information corresponding to the color image through the depth camera 130 . In step S320, the processor 110 may perform image space cutting and image rotation processing according to the color image and the depth information, so as to generate a plurality of depth images. In step S330, the processor 110 may input a plurality of depth images to the environment image recognition module 121, so that the environment image recognition module 121 outputs the displacement coordinate parameters. In this embodiment, the environment image recognition module 121 may include a neural network (Neural network) computing model, and the environment image recognition module 121 may be pre-trained, for example, to learn to recognize obstacles in the depth image, and it may be based on the depth The recognition result of the image is used to generate the displacement coordinate parameter. In step S340, the environment image recognition module 121 can output the displacement coordinate parameter to the robot arm control module 122, so that the robot arm control module 122 can control the movement of the robot arm 140 according to the displacement coordinate parameter. Therefore, the automatic control system 100 of this embodiment can effectively control the robot arm 140 . Moreover, the detailed implementation of each step will be described in detail by the following multiple embodiments.

FIG. 4 is a schematic diagram of multiple modules of a neural network operation according to an embodiment of the present invention. Fig. 5 is a flowchart of an automatic control method according to another embodiment of the present invention. Referring to FIG. 1 , FIG. 4 and FIG. 5 , the automatic control system 100 may execute steps S510 - S570 to operate the robot arm 140 . In this embodiment, the memory 120 can also store an image processing module 411 , a track recording module 412 , a feedback module 413 and an output module 414 . The processor 110 can execute the environment image recognition module 121 and the machine Manipulator control module 122 , image processing module 411 , track recording module 412 , feedback module 413 and output module 414 . It should be noted that the automatic control system 100 can execute the image processing module 411 , the environment image recognition module 121 , the output module 414 and the robot arm control module 122 to control the robot arm 140 . The image processing module 411 , the environment image recognition module 121 , the output module 414 and the robot arm control module 122 may belong to or be integrated into the control module 410 . Moreover, in one embodiment, the automatic control system 100 can enter a training mode to train the environment image recognition module 121 . In this regard, the automatic control system 100 can execute the image processing module 411, the environment image recognition module 121, the track record module 412, the feedback module 413, the output module 414 and the robot arm control module 122 to train the environment image recognition module. Group 121. The image processing module 411 , the environment image recognition module 121 , the trajectory recording module 412 , the feedback module 413 , the output module 414 and the robot arm control module 122 may belong to or be integrated into the training module 420 .

FIG. 6 is a schematic diagram of a color image according to an embodiment of the invention. FIG. 7 is a schematic diagram of a plurality of depth images according to an embodiment of the invention. This will be further described below with reference to FIG. 6 and FIG. 7 . In this embodiment, the automatic control system 100 may further include an input module, such as a mouse, a keyboard and other input devices, and may receive user input data (or set parameters). In step S510, the processor 110 may set a starting position parameter and a target position parameter according to the received input data. In step S520 , the processor 110 can obtain the color image 600 shown in FIG. 6 of the current frame and the depth information corresponding to the color image 600 through the depth camera 130 . Color image 600 may include surgical site image 610 as well as environment image 620. The processor 110 can define a target position 630 in the color image 600 according to the target position parameter, and a range 601 of the target position 630 . In step S530 , the processor 110 may execute the image processing module 411 to perform image space cutting and image rotation processing according to the color image 600 and depth information, so as to generate a plurality of depth images 700_1 - 700_16 as shown in FIG. 7 .

For example, the image processing module 411 may perform RGB digital image space segmentation on the color image 600 to increase the diversity of environmental features, and generate depth images 700_1 , 700_5 , 700_9 , and 700_13 respectively corresponding to different depths. Next, the image processing module 411 can respectively rotate the depth images 700_1, 700_5, 700_9, and 700_13, such as 90 degrees, 180 degrees, and 270 degrees, to further generate multiple depth images 700_2~700_4, 700_6~700_8, 700_10~ 700_12, 700_14~700_16, and increase the sample data. From another point of view, there are 16 sample data for each pixel position in the image. In other words, the automatic control system 100 can process a color image obtained by the depth camera 130 in each frame, and generate a plurality of corresponding depth images 700_1~700_16, so as to effectively analyze the current space environment at each moment in three dimensions. state. In addition, the number of depth images of the present invention is not limited to FIG. 7 .

In step S540 , the processor 110 inputs a plurality of depth images to the environment image recognition module 121 , so that the environment image recognition module 121 outputs the displacement coordinate parameters to the output module 414 . In this embodiment, the environment image recognition module 121 can execute the neural network calculation, and the environment image recognition module 121 can identify effective safe space images from the depth images 700_1~700_16 to select the depth image One of 700_1~700_16 is used as an effective safe space image, and the displacement coordinate parameter is further determined according to the effective safe space image. In other words, the environment image recognition module 121 can three-dimensionally determine a safer movement path in the space of the current frame, so as to provide corresponding displacement coordinate parameters to the output module 414 .

In more detail, after each of the depth images 700_1~700_16 is input to the neural network computing model of the environment image recognition module 121, the neural network computing model of the environment image recognition module 121 can analyze each of the depth images 700_1~700_16. An eigenvalue (environmental effective space eigenvalue) analysis is performed on a plurality of pixels to obtain the eigenvalue weight of each pixel, wherein the eigenvalue analysis is used to determine the object evaluation of each pixel. Therefore, the environment image recognition module 121 can, for example, generate a plurality of spatial weight matrix data corresponding to the respective depth images 700_1 - 700_16 . In this regard, the environment image recognition module 121 can perform neural network calculations according to the plurality of spatial weight matrix data corresponding to the depth images 700_1 to 700_16 to determine effective safe space images.

For example, the neural network computing model of the environment image recognition module 121 can determine the safe displacement direction and position of the next frame according to the current position of the robotic arm 140 in each of the depth images 700_1 - 700_16 . For example, the weights belonging to objects (obstacles or surgical sites) in the depth images 700_1˜700_16 may be higher. The environment image recognition module 121 can determine the one with the lowest weight value corresponding to each pixel within the unit movement distance around the current position of the robot arm 140 in each of the depth images 700_1~700_16 (and move toward the target position), as The robot arm 140 is at the position of the next frame, and uses the corresponding depth image as an effective safe space image. Therefore, the automatic control system 100 can drive the mechanical arm 140 towards this position Move, so as to effectively avoid contact or collision between the robotic arm 140 and objects (obstacles or surgical sites).

In step S550, the output module 414 can output the displacement coordinate parameter to the robot arm control module 122, so that the robot arm control module 122 can control the movement of the robot arm 140 according to the displacement coordinate parameter. In this embodiment, the output module 414 can further output the movable direction information and movable position information for the robot arm 140 to the robot arm control module, for example, according to the analysis and calculation results of the environment image recognition module 121 122. In step S560 , the processor 110 may return the current end coordinate parameters of the robotic arm 140 (such as the coordinates of the gripper 141 of the robotic arm 140 shown in FIG. 2 ) through the robotic arm control module 122 . In step S570, the processor 110 may determine whether the robot arm 140 has reached the target position. If yes, the automatic control system 100 ends the current control task. If not, the automatic control system 100 re-executes steps S510-S570 to determine the next displacement direction and position of the mechanical arm 140 according to the next frame of color image and its depth information provided by the depth camera 130 . Therefore, the automatic control system 100 and the automatic control method of this embodiment can effectively control the movement of the robotic arm 140 in space through visual image control, and avoid objects (obstacles) in the process of moving the robotic arm 140 in space. object or surgical site) contact or collision.

FIG. 8 is a flowchart of a training environment image recognition module according to an embodiment of the present invention. FIG. 9 is a schematic diagram of a plurality of effective safe space images according to an embodiment of the present invention. Referring to Fig. 1, Fig. 4, Fig. 5, Fig. 8 and Fig. 9, the automatic control system 100 can execute steps S510~S570, S810, S820 to train the environment image recognition The neural network calculation model in the recognition module 121. The automatic control system 100 can enter the training mode to execute the training module 420 . In this embodiment, the automatic control system 100 can also perform steps S510-S570 sequentially as in the flow of the above-mentioned embodiment in FIG. After parameterizing (after step S560 above), the automatic control system 100 may execute step S810. In step S810, the processor 110 can execute the trajectory recording module 412, so as to record the displacement direction to the effective safe space image 800_1 shown in FIG. 9 according to the current terminal coordinate parameter and the previous terminal coordinate parameter through the trajectory recording module 412, wherein The effective safe space image 800_1 includes a movement track 900_1 of the robot arm 140 .

Next, in step S820, the processor 110 may execute the feedback module 413 to calculate the distance parameter between the current terminal coordinate parameter and the target position through the feedback module 413, and train the environment image recognition module 121 according to the distance parameter. For example, the processor 110 can determine whether the result of moving the robot arm 140 makes the robot arm 140 move toward the target position (whether the distance between the robot arm 140 and the target position is shortened), and define whether the current movement is appropriate, and then Feedback training the neural network computing model in the environment image recognition module 121 . Finally, the processor 110 may continue to execute step S570. Moreover, the automatic control system 100 can, for example, repeatedly analyze the shooting results of multiple frames of the depth camera 130 for a continuous period of time to generate a plurality of continuous effective safe space images 800_1~800_N as shown in FIG. 9 , wherein N is a positive integer greater than 1. It is worth noting that the effective safe space images 800_1~800_N may respectively include the mechanical arm 140 of the previous multiple frames accumulated and recorded in time sequence The movement trajectories 900_1~900_N formed by the multiple displacement positions of . In other words, the neural network computing model in the environmental image recognition module 121 can not only effectively identify objects in the image, but also effectively train the neural network computing model in the automatic control system 100. The output result is to drive the mechanical arm 140 toward The target position is moved and an optimal path is selected, so that each movement of the robot arm 140 is not simply to avoid obstacles in the environment.

To sum up, the automatic control method and automatic control system of the robotic arm of the present invention can enable the neural network computing model in the environmental image recognition module to learn to judge objects in the image and learn to make each time The displacement coordinate parameters of the neural network calculation result can make the robot arm move towards the target position. Therefore, the automatic control method and the automatic control system of the present invention can effectively control the movement of the robot arm to the target position, and can effectively avoid obstacles in the environment during the movement of the robot arm.

In order to make the content of the present invention more comprehensible, the following specific examples are given as examples in which the present invention can indeed be implemented. In addition, wherever possible, elements/components/steps using the same reference numerals in the drawings and embodiments represent the same or similar parts.

100: Automatic control system

110: Processor

120: memory

121:Environmental image recognition module

122: Robotic arm control module

130: Depth camera

140: Mechanical arm

Claims (16)

An automatic control method of a mechanical arm, comprising: obtaining a color image and a depth information corresponding to the color image through a depth camera; performing an image space cutting process and an image rotation process according to the color image and the depth information, to generating a plurality of depth images, including: cutting the color image in digital image space to generate a plurality of cutting depth images; and rotating the cutting depth images respectively to generate the depth images; inputting the depth images To an environment image recognition module, so that the environment image recognition module outputs a displacement coordinate parameter according to the depth images, including: performing eigenvalue analysis on each of the depth images to obtain a corresponding spatial weight matrix data; and judging Each of the space weight matrix data with the lowest value is used as the position of the robot arm in the next frame, and the corresponding depth images are used as an effective safe space image; and the displacement coordinate parameters are output to a robot arm control module, The robot arm control module controls the robot arm to move towards the spatial weight matrix data having the lowest value according to the displacement coordinate parameter. The automatic control method as described in claim 1 further includes: setting a starting position parameter and a target position parameter, wherein the starting position is an end position parameter corresponding to the robot arm. The automatic control method according to claim 2, wherein the image space cutting process and the image rotation process are executed according to the target position parameter or a position parameter of an obstacle. The automatic control method as described in claim 2, wherein the environment image recognition module is used to execute a neural network calculation, and the environment image recognition module is used to recognize the effective safe space image from the depth images, and according to The effective safe space image is used to determine the displacement coordinate parameter. The automatic control method as described in claim 4 further includes: after the robot arm moves according to the displacement coordinate parameter, returning a current end coordinate parameter of the robot arm through the robot arm control module. The automatic control method as described in claim item 5, wherein the step of inputting these depth images to the environment image recognition module further includes: executing a track record module, so as to use the track record module according to the current end coordinate parameter and a previous end coordinate parameter records a displacement direction to the effective safe space image; and executes a feedback module to calculate a distance parameter between the current end coordinate parameter and the target position through the feedback module, and according to The distance parameter is used to train the environment image recognition module. The automatic control method as described in claim 4, wherein the step of inputting the depth images to the environment image recognition module includes: analyzing the depth images through the environment image recognition module to generate corresponding A plurality of spatial weight matrix data of the depth images; and performing the neural network calculation according to the spatial weight matrix data corresponding to the depth images through the environment image recognition module to determine the effective safe space image. The automatic control method as described in claim 7, wherein the step of outputting the displacement coordinate parameter to the robot arm control module includes: executing an output module, so as to use the output module according to the analysis and analysis of the environmental image recognition module The calculation result further outputs a movable direction information and a movable position information for the robot arm to the robot arm control module. An automatic control system of a mechanical arm, comprising: a depth camera, used to obtain a color image and a depth information corresponding to the color image; and a processor, coupled to the mechanical arm and the depth camera, and used to The color image and the depth information are subjected to an image space slicing process and an image rotation process to generate multiple depth images, wherein the processor performs digital image space slicing on the color image to generate multiple cutting depth images, and the The processor rotates the cutting depth images respectively to generate the depth images, wherein the processor inputs the depth images to an environment image recognition module so that the environment image recognition module can use the depth images Outputting a displacement coordinate parameter, the processor performs eigenvalue analysis on each of the depth images to obtain a corresponding spatial weight matrix data, and the processor determines each of the spatial weight matrices with the lowest value The data is used as the position of the robot arm in the next frame, and the corresponding depth images are used as an effective safe space image, and the processor outputs the displacement coordinate parameters to a robot arm control module, so that the robot arm can control The module controls the robot arm to move towards the spatial weight matrix data with the lowest value according to the displacement coordinate parameter. The automatic control system according to claim 9, wherein the processor sets an initial position parameter and a target position parameter, wherein the initial position corresponds to an end position parameter of the robot arm. The automatic control system as claimed in claim 10, wherein the processor executes the image space cutting process and the image rotation process according to the target position parameter or a position parameter of an obstacle. The automatic control system according to claim 10, wherein the environment image recognition module is used to execute a neural network operation, and the processor executes the environment image recognition module to recognize the effective safe space from the depth images image, and determine the displacement coordinate parameter according to the effective safe space image. The automatic control system according to claim 12, wherein after the robot arm moves according to the displacement coordinate parameter, the robot arm control module returns a current end coordinate parameter of the robot arm. The automatic control system as described in claim 13, wherein the processor executes a trajectory recording module, so as to record a displacement direction to the effective safety according to the current terminal coordinate parameter and a previous terminal coordinate parameter through the trajectory recording module spatial image, and the processor executes a feedback module to calculate A distance parameter between the current end coordinate parameter and the target position is calculated, and the environment image recognition module is trained according to the distance parameter. The automatic control system as claimed in claim 12, wherein the processor analyzes the depth images through the environment image recognition module to generate a plurality of spatial weight matrix data corresponding to the depth images, and the processor uses the The environment image recognition module performs the neural network calculation according to the spatial weight matrix data corresponding to the depth images to determine the effective safe space image. The automatic control system as described in claim 15, wherein the processor executes an output module, so as to further output a movable for the robot arm through the output module based on the analysis and calculation results of the environmental image recognition module Direction information and a movable position information are sent to the robot arm control module.

Images