CN115590454A

CN115590454A - Endoscope control state automatic switching method and device, equipment and storage medium

Info

Publication number: CN115590454A
Application number: CN202211602932.4A
Authority: CN
Inventors: 何进雄; 谭有余; 谭文
Original assignee: Zhuhai Seesheen Medical Technology Co ltd
Current assignee: Zhuhai Seesheen Medical Technology Co ltd
Priority date: 2022-12-14
Filing date: 2022-12-14
Publication date: 2023-01-13
Anticipated expiration: 2042-12-14
Also published as: CN115590454B

Abstract

The invention belongs to the technical field of endoscope control, and discloses a method, a device, equipment and a storage medium for automatically switching an endoscope control state, wherein when an endoscope is in an automatic moving state and moves to any observation node on a preset planning path, the endoscope is controlled to acquire image data at the observation node; and then, identifying a current three-dimensional space diagram according to the image data, and if the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node is smaller than a matching threshold value, switching the control state of the endoscope from an automatic moving state to an automatic stopping state, so that the control state of the endoscope can be automatically switched by sensing the actual cavity environment of each observation node in the process that the endoscope automatically moves on the way of a preset planning path, thereby improving the intelligent degree and the safety performance.

Description

Endoscope control state automatic switching method and device, equipment and storage medium

Technical Field

The invention belongs to the technical field of endoscope control, and particularly relates to an endoscope control state automatic switching method, an endoscope control state automatic switching device, endoscope control state automatic switching equipment and an endoscope control state automatic switching storage medium.

Background

Laparoscopic surgery is a type of minimally invasive surgical procedure in which a long-handled instrument is inserted through a small incision into a patient's body at a target site requiring surgery. Compared with the traditional open type operation, the laparoscopic operation has small wound and less adhesion, and can reduce the pain of patients and shorten the recovery time. Conventional manual laparoscopic surgery suffers from a number of limitations, including inability to sense depth, poor control over the camera, limited angle and space in which the instrument tip can freely rotate, and limited range of motion of the surgeon's surgical instruments. These limitations can lead to the surgeon suffering from an unnatural and painful surgical position during the procedure, which can be prone to fatigue.

With the rapid development of medical technology, laparoscopic surgery assisted robots (surgical robots for short), such as the da vinci system, are widely used in various medical surgeries. The surgical robot comprises a control console, a robot mechanical arm and imaging processing equipment, a doctor can drive the robot mechanical arm to drive the endoscope to perform surgical operation through the control console, and the robot mechanical arm can also automatically drive the endoscope to reach a focus position (namely a target position needing to be operated) through an automatic control program. The three-dimensional visualization function of the surgical robot can provide depth perception, and the three-dimensional visualization function of the surgical robot is similar to a miniaturized mechanical arm of a wrist joint, so that the flexibility and the motion range of a surgeon are improved. In actual operation, a three-dimensional image of a focus position needs to be constructed through a preoperative CT image or an MRI image, an optimal operation path is planned, and the robot manipulator controls the endoscope to reach the focus position based on the planned path. The endoscope is in an automatic moving state when moving on a planned path automatically, and is automatically switched to an automatic stopping state after reaching a focus position so as to facilitate further operation.

However, although the results of human tissues and organs can be clearly obtained through the CT image and MRI image recognition methods, the characteristics such as the real color and properties of mucosal tissues in the body lumen cannot be represented, and the inflammation and congestion degree of some tissues cannot be completely represented. Therefore, some lesion problems of the mucosa tissue of the lumen may be missed based on the planned path of the CT image or the MRI image. That is, when planning a path preoperatively in the prior art, it is generally assumed that no other lesions exist on the path other than the identified target lesion. In fact, due to the lesion problem of the mucosa tissue of the lumen, abnormal biological tissues such as polyps or cyst of mucosa may be generated on the planned path. At this time, if the endoscope is forcibly moved past, the endoscope may touch the abnormal biological tissue, which may cause an operation accident.

Aiming at the situation, in the prior art, the endoscope can only be identified by the eyes of a doctor in an operation, and the endoscope is switched into a manual operation mode through manual intervention, so that the efficiency is too low, mistakes are easily made, and the energy of the doctor is wasted. Therefore, the existing endoscope automatic control method can only automatically switch the control state of the endoscope when reaching the preset destination of the planned path, namely the focus position, but cannot sense the actual cavity environment in the process of planning the path to automatically switch the control state of the endoscope, so that the intelligent degree is low, and the safety performance is influenced.

Disclosure of Invention

The invention aims to provide an endoscope control state automatic switching method, an endoscope control state automatic switching device, endoscope control state automatic switching equipment and an endoscope storage medium, which can sense the actual cavity environment in the process of planning a path to automatically switch the control state of an endoscope, so that the intelligent degree is improved, and the safety performance is improved.

The invention discloses a method for automatically switching the operation state of an endoscope, which comprises the following steps:

when the endoscope is in an automatic moving state and moves to any observation node on a preset planning path, controlling the endoscope to acquire and obtain first image data at the observation node;

constructing a current three-dimensional space map according to the first image data;

calculating the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node;

and if the matching degree is smaller than a matching threshold value, switching the control state of the endoscope from the automatic moving state to an automatic stopping state.

In some embodiments, after switching the manipulation state of the endoscope from the automatic moving state to the automatic staying state, the method further comprises:

when the duration of the endoscope in the automatic stopping state reaches a specified duration, controlling the endoscope to acquire second image data;

constructing a target three-dimensional space map according to the second image data;

comparing the target three-dimensional space diagram with the preset three-dimensional space diagram to obtain the position and the shape of the obstacle;

planning an obstacle avoidance channel based on the target three-dimensional space diagram according to the obstacle position and the obstacle form;

and if the space size of the obstacle avoidance channel is enough for the endoscope to pass through without obstacles and move to the next observation node, switching the control state of the endoscope from the automatic stay state to the automatic movement state.

In some embodiments, controlling the endoscope to acquire second image data comprises:

comparing the current three-dimensional space map with the preset three-dimensional space map to obtain an obstacle area;

adjusting the posture of the endoscope according to the obstacle area;

and controlling the endoscope to acquire second image data by adopting the adjusted posture.

In some embodiments, comparing the target three-dimensional space map with the preset three-dimensional space map to obtain the position and shape of the obstacle includes:

comparing the target three-dimensional space map with the preset three-dimensional space map to obtain a distinguishing map domain;

identifying a plurality of positions to be selected according to the distinguishing graph field;

determining a candidate position located in the obstacle area as an obstacle position;

and carrying out image recognition on the specified neighborhood taking the barrier position as the center to obtain the barrier shape.

In some embodiments, after controlling the endoscope to acquire the second image data, the method further comprises:

fusing the first image data and the second image data to obtain target image data;

and constructing a target three-dimensional space map according to the second image data, wherein the method comprises the following steps:

and constructing a target three-dimensional space map according to the target image data.

The second aspect of the present invention discloses an endoscope manipulation state automatic switching apparatus, including:

the first acquisition unit is used for controlling the endoscope to acquire and obtain first image data at an observation node when the endoscope is in an automatic moving state and moves to any observation node on a preset planning path;

the first construction unit is used for constructing a current three-dimensional space map according to the first image data;

the computing unit is used for computing the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node;

and the switching unit is used for switching the control state of the endoscope from the automatic moving state to the automatic stopping state when the matching degree calculated by the calculating unit is smaller than the matching threshold.

In some embodiments, further comprising:

the second acquisition unit is used for controlling the endoscope to acquire and obtain second image data when the duration of the endoscope in the automatic stopping state reaches a specified duration;

the second construction unit is used for constructing a target three-dimensional space map according to the second image data;

the identification unit is used for comparing the target three-dimensional space diagram with the preset three-dimensional space diagram to obtain the position and the form of the obstacle;

the obstacle avoidance planning unit is used for planning an obstacle avoidance channel based on the target three-dimensional space map according to the obstacle position and the obstacle form;

the switching unit is further configured to switch the control state of the endoscope from the automatic stay state to the automatic moving state when the space size of the obstacle avoidance channel is sufficient for the endoscope to pass through without obstacles and move to a next observation node.

In some embodiments, the second acquisition unit comprises:

the comparison subunit is used for comparing the current three-dimensional space map with the preset three-dimensional space map when the duration of the endoscope in the automatic stopping state reaches a specified duration to obtain an obstacle area;

an adjustment subunit configured to adjust a posture of the endoscope in accordance with the obstacle region;

and the acquisition subunit is used for controlling the endoscope to acquire second image data by adopting the adjusted posture.

A third aspect of the invention discloses an electronic device comprising a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory for executing the endoscope manipulation state automatic switching method disclosed in the first aspect.

A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the endoscope manipulation state automatic switching method disclosed in the first aspect.

The method, the device, the equipment and the storage medium for automatically switching the endoscope control state have the advantages that when the endoscope is in the automatic moving state and moves to any observation node on the preset planning path, the endoscope is controlled to acquire and obtain image data at the observation node; and then, identifying a current three-dimensional space diagram according to the image data, and if the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node is smaller than a matching threshold value, switching the control state of the endoscope from an automatic moving state to an automatic stopping state, so that the control state of the endoscope can be automatically switched by sensing the actual cavity environment of each observation node in the process that the endoscope automatically moves on the way of a preset planning path, thereby improving the intelligent degree and the safety performance.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles and effects of the invention.

Unless otherwise specified or defined, the same reference numerals in different figures refer to the same or similar features, and different reference numerals may be used for the same or similar features.

FIG. 1 is a flow chart of a method for automatically switching the operating state of an endoscope;

FIG. 2 is a schematic structural view of an endoscope manipulation state automatic switching apparatus;

fig. 3 is a schematic structural diagram of an electronic device.

Description of reference numerals:

201. a first acquisition unit; 202. a first building element; 203. a calculation unit; 204. a switching unit; 205. a second acquisition unit; 206. a second building element; 207. an identification unit; 208. an obstacle avoidance planning unit; 301. a memory; 302. a processor.

Detailed Description

Unless specifically stated or otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In the case of combining the technical solutions of the present invention in a realistic scenario, all technical and scientific terms used herein may also have meanings corresponding to the purpose of achieving the technical solutions of the present invention. As used herein, "first and second" \ 8230, "are used merely to distinguish between names and do not denote a particular quantity or order. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

As used herein, unless otherwise specified or defined, the terms "comprises," "comprising," and "comprising" are used interchangeably to refer to the term "comprising," and are used interchangeably herein.

It is needless to say that technical contents or technical features which are contrary to the object of the present invention or clearly contradicted by the object of the present invention should be excluded. In order to facilitate an understanding of the invention, specific embodiments thereof will be described in more detail below with reference to the accompanying drawings.

As shown in fig. 1, an embodiment of the present invention discloses an endoscope operating state automatic switching method, an execution main body of the method may be an electronic device such as a computer, a notebook computer, a tablet computer, or the like, or an endoscope operating state automatic switching apparatus embedded in the electronic device, which is not limited in the present invention. In this embodiment, an electronic device is taken as an example for explanation. The method comprises the following steps S10-S40:

and S10, when the endoscope is in an automatic moving state and moves to any observation node on a preset planning path, the electronic equipment controls the endoscope to acquire and obtain first image data at the observation node.

In the embodiment of the present invention, before step S10 is executed, the electronic device may first construct an in-vivo control environment three-dimensional model as a target three-dimensional model according to a CT image or an MRI image taken before an operation, then perform path planning before the operation/simulation according to the target three-dimensional model, obtain a preset planned path of the endoscope, and then obtain a plurality of observation nodes located on the preset planned path of the endoscope.

Specifically, medical images such as CT images or MRI images of the lesion area may be acquired, a VTK algorithm may be used to reconstruct a three-dimensional image of the medical images, a target three-dimensional model of the lesion area and corresponding three-dimensional model information may be acquired, and a preset planning path from a starting point to an end point (i.e., a lesion position) may be generated in the target three-dimensional model based on the three-dimensional model information. It should be noted that, based on the three-dimensional model information, one or more lesions inside the human body may be identified, and when a plurality of lesions are identified, that is, a plurality of lesion positions (e.g., B, C, D) exist, a plurality of preset planning paths may be generated, such as starting point a to lesion position B, starting point a to lesion position C, starting point a to lesion position D, starting point B to lesion position C, or starting point C to lesion position D. That is, a certain lesion position is generally determined as an end point, and a lesion is located on a preset planned path except the end point (or there is a possibility that both the start point and the end point are located), and it is assumed that no other lesion is located on the way of the path.

As an optional implementation, a specific implementation of the electronic device planning the path may include:

acquiring a binarization image sequence of a patient according to a CT image of the patient; carrying out interval processing on the binary image sequence to establish a binary three-dimensional grid map with pixels as basic grid units; then obtaining a designated starting point and a focus position, and obtaining a motion path from the designated starting point to the focus position in a binary three-dimensional grid map according to a path cost algorithm (including an A-star algorithm superposition cost function); and finally, smoothing the motion path to obtain a preset planning path. As an alternative embodiment, taking a section of trachea as an example, the step of the electronic device executing the path cost algorithm may include steps 110 to 140, specifically the following steps:

and 110, constructing a three-dimensional grid map according to the binary image sequence of the CT image of the section of the trachea.

Specifically, the values of F, G and H of all nodes in the three-dimensional grid graph are set to 0, where F, G and H correspond to the function values of F (n), G (n) and H (n), respectively, F (n) is the whole cost function and represents the cost estimate of the minimum cost path, and G (n) is a known cost function and represents the known cost from the starting node to the current node; h (n) is an estimated cost function representing the estimated cost from the current node to the target node.

And step 120, constructing an Open list and a Closed list, and executing initialization of the Open list and the Closed list.

Step 130, constructing a start node (specifying a start location) and a target node (specifying an end location).

Specifically, F, G, and H values of the start node are calculated, and then the start node is placed in an Open list, and the start node is regarded as the current node.

And 140, starting a path searching step for the current node until a target node is searched or the Open list is empty, and exiting the path searching step.

Specifically, the step of searching the path initiated by the current node includes steps 141 to 143, and the specific steps are as follows:

step 141, performing first loop judgment to judge whether the coordinates of the current node are the coordinates of the target node; if yes, the search is successful, and a search path is obtained. If not, searching the node with the minimum F value in the Open list as a new current node.

And step 142, performing second loop judgment to judge whether a node exists in the Open list.

Specifically, if the Open list is empty, the second loop judgment is exited, the path search fails, and the path search is ended; and if the Open list is not empty, putting the current node into the Closed list, and taking the node with the minimum F value found in the Open list as a new current node.

And 143, sequentially traversing 26 adjacent nodes of the current node, and judging the positions of the adjacent nodes.

Specifically, a maximum of 26 adjacent nodes exist in a node in the 3D space. And judging whether the adjacent node is positioned in the Open list or not only when the adjacent node of the current node is a non-obstacle node and is not positioned in the Closed list.

If the adjacent node is not in the Open list, adding the adjacent node into the Open list, calculating and storing the values of F, G and H of the adjacent node, setting the current node as the parent node of the adjacent node, and then jumping to the step 142;

if the adjacent node is in the Open list, recalculating F, G and H values of the adjacent node, and comparing the recalculated F value with the F value before calculation:

if the recalculated F value is smaller than the F value before calculation, the recalculated F, G, H values are used as the F, G, H values of the adjacent node, the current node is set as the parent node of the adjacent node, and then the step 143 is skipped;

if the calculated F value is greater than the F value before calculation, the F, G, and H values of the adjacent node and the parent node corresponding to the adjacent node are not updated, and then the process goes to step 143.

And circularly operating the steps 141-143 until the loop is tripped.

Specifically, the conditions for the jump-out cycle are as follows: if the Open list is empty, the path search fails, and a proper path cannot be found; if the coordinates of a certain node in the Open list are equal to the coordinates of the target node, the path search is successful, and the search path can be obtained. It can be understood that, in the process of traversing nodes, when an adjacent node closest to the current node is found, the current node is set as a parent node of the adjacent node, in the process of traversing, F, G, and H values of the adjacent node and corresponding parent node information are stored, and in the process of loop search, when a certain node coordinate is the same as a target node coordinate in an Open list, a node marked as a parent node can trace back from the target node until the node returns to the start node to obtain the whole path, so as to generate a planned path.

After the preset planned path is generated, the preset planned path can be uniformly divided into N +1 sub-paths, and the boundary point between every two sub-paths is set as an observation node except the starting point and the end point, so that N observation nodes are obtained. Wherein the observation node includes but is not limited to a time node or a location node. The distance between every two adjacent observation nodes can be limited by setting a threshold value according to actual conditions, so that frequent operation of the endoscope caused by too small distance between the observation nodes can be avoided, and low identification accuracy caused by too large distance between the observation nodes can also be avoided.

The moving mode of the endoscope on the preset planning path can comprise a manual control moving mode or an automatic control mode, under the automatic control mode, the state of the endoscope mainly comprises an automatic moving state or an automatic stopping state, and when the endoscope is positioned in the cavity channel, the endoscope is in the automatic moving state so as to execute corresponding control instructions such as moving speed, moving steering, posture adjustment and the like set under the control state; when the endoscope reaches the focus position, the endoscope is automatically switched to an automatic stay state so as to execute corresponding control instructions of posture adjustment, image shooting and the like set in the control state, so that the operation is further facilitated.

Wherein the endoscope starts an automatic movement state from a start point, and moves through each observation node one by one, and finally reaches an end point. Each time the endoscope reaches any observation node

And controlling the endoscope to acquire and obtain first image data at the observation node.

Wherein, the endoscope can be provided with image sensors such as CCD, CMOS or CIS, etc., to realize image acquisition function. As a preferred embodiment, the endoscope may further be provided with an infrared sensor, and the endoscope acquires image data at the observation node by using the infrared sensor to obtain the first image data. The infrared sensor is adopted for collecting image data, so that the influence of complex light rays, mucus and the like of the environment in a human body on the sight line can be avoided, and the accuracy of the image data is improved.

And S20, the electronic equipment constructs a current three-dimensional space map according to the first image data.

In the embodiment of the present invention, N observation nodes may respectively correspond to N preset three-dimensional space maps, where the preset three-dimensional space maps are used to represent the cavity environment of cavity road segments (such as straight cavity road segments, multi-intersection bifurcation road segments, or curved cavity road segments) located near each observation node. The preset three-dimensional space map can be generated by segmentation in a target three-dimensional model constructed in advance based on the position of each observation node.

Therefore, when the endoscope moves to a certain observation node, three-dimensional reconstruction can be performed based on the first image data acquired at the observation node position, and a current three-dimensional space map corresponding to the current observation node is obtained.

And S30, the electronic equipment calculates the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node.

The preset three-dimensional space map corresponding to each observation node is generated in a simulation mode under the condition that no other focus exists, and therefore abnormal biological tissue information, such as abnormal biological tissue (such as mucosal polyp or mucosal cyst) information generated due to the lesion of the cavity and canal mucosal tissue, is not included. The preset three-dimensional space graph can correspond to an atlas, and the atlas comprises a specified number of sub-sample graphs.

In step S30, the electronic device may perform enhancement sharpening on the acquired current three-dimensional space image to obtain a first processed image, then segment the enhanced sharpened first processed image into a specified number of sub-space images, compare each sub-space image with a corresponding sub-sample image in an image set corresponding to the preset three-dimensional space image, and obtain a matching degree between the current three-dimensional space image and the preset three-dimensional space image based on a comparison result.

The implementation method of performing, by the electronic device, an enhanced sharpening process on the acquired current three-dimensional spatial map to obtain the first processed image may specifically include:

the method comprises the steps of carrying out illumination estimation on a current three-dimensional space diagram based on a Candy algorithm, converting the illumination estimation into a fine reflection component in a Log field through the following formula (1), and finally carrying out quantization processing to obtain an enhanced sharpened first processed image.

R(x，y) = ln I(x，y) - ln[G(x，y)*I(x，y)] （1）

Where R (x, y) is the output first processed image, i.e., the reflection component; i (x, y) is the current three-dimensional space diagram of the input; g (x, y) is a Canny operator-based weighted guided filtering factor, and the illumination is estimated as an environment function; * Is a convolution operation.

The algorithm process specifically comprises the steps that the electronic equipment firstly inputs an original current three-dimensional space diagram I (x, y) and a filtering radius range, then calculates a result of Gaussian filtering of the current three-dimensional space diagram I (x, y) according to a weighting guide filtering factor G (x, y), obtains a filtering diagram S (x, y) = ln [ G (x, y) × I (x, y) ], then calculates a calculation result ln R (x, y) according to a formula (1), and finally quantizes the calculation result into a pixel value in a range of [0, 255], and obtains an output first processed image R (x, y).

Specifically, for each subspace graph, if the similarity between a certain subsample graph and the subspace graph reaches a similarity threshold value through comparison in a graph set corresponding to a preset three-dimensional space graph, the two graphs are judged to be overlapped, the number of overlapped graphs is recorded, and the number of overlapped graphs is cumulatively added by one, and the next subspace graph is continuously compared. And finally, when the comparison of all the sub-space maps is finished, counting the current coincidence quantity, and calculating according to the current coincidence quantity to obtain the matching degree of the current three-dimensional space map and the preset three-dimensional space map.

For example, the ratio of the current coincidence number to the number of the subspace maps (i.e., the designated number) is multiplied by 100% as the matching degree of the current three-dimensional space map and the preset three-dimensional space map.

Preferably, in the process of comparing each subspace map, a PHash algorithm may be adopted, which first transforms the picture from a pixel domain to a frequency domain by using Discrete Cosine Transform (DCT), and then retains elements in an upper left corner region of a frequency coefficient matrix to calculate an image hash value as a similarity, so that more image details can be retained, and the calculation accuracy is improved.

Specifically, for each subspace graph, the subspace graph is reduced to an n × n reduction graph to obtain n ² A plurality of pixels; then, converting the miniature image of n multiplied by n into a gray image, marking as M (x, y), carrying out DCT (discrete cosine transformation) change on the gray image, taking a k multiplied by k matrix at the upper left corner of a coefficient matrix, calculating the DCT average value of each pixel in the k multiplied by k matrix, comparing the DCT value of each pixel in the ergodic gray image with the DCT average value, and marking as 1 if the DCT average value is greater than or equal to the DCT average value; if the average value is smaller than the DCT average value, marking as 0; thereby generating a binary array as a hash value of the subspace map. Likewise, the hash value of the sub-sample map corresponding to the subspace map may be computed in this way. And finally, calculating the Hamming distance between the hash value of the subspace graph and the hash value of the corresponding sub sample graph, and determining the similarity of the Hamming distance and the sub sample graph according to the Hamming distance. Wherein, the smaller the Hamming distance is, the more similar the two are shown, and the larger the Hamming distance is, the larger the difference between the two is shown. For example, the reciprocal of the hamming distance can be taken as the similarity of the two.

In information coding, the number of bits coded differently on the corresponding bits of two legal codes is called the code distance, also called hamming distance. In an active code set, the minimum of the hamming distances of any two codewords is called the hamming distance of the code set. Examples are as follows: 10101 and 00110 are different from the first digit to the fourth digit and the fifth digit in sequence, the Hamming distance is 3.

And S40, if the matching degree of the current three-dimensional space diagram and the preset three-dimensional space diagram is smaller than a matching threshold, the electronic equipment switches the control state of the endoscope from an automatic moving state to an automatic stopping state.

It can be understood that, also corresponding to the observation node, the preset three-dimensional space map represents the lumen environment around the observation node, the observation node is located between the starting point and the end point of the preset planned path and is a non-focal target, the current three-dimensional space map represents the actual lumen environment for the endoscope to reach the observation node or is reconstructed from the image taken by the endoscope, and if the current three-dimensional space map is not matched with the preset three-dimensional space map, it may be suspected that abnormal biological tissues generated due to lesion of the lumen mucosal tissues may exist in the lumen environment in the range near the observation node. In this case, the electronic device switches the manipulation state of the endoscope from the automatic movement state to the automatic stay state.

It can be understood that the plurality of observation nodes arranged on the preset planning path are only used as marking nodes, the positions of the lesions are not on the preset planning path, when the endoscope preoperative automatic control planning is performed, the automatic control command corresponding to the observation nodes (the positions of the planning end points are not reached) is not generally used for stay observation, and when the current three-dimensional space diagram obtained by the real-time image reconstruction and obtained by the observation nodes is not matched with the preset three-dimensional space diagram, the electronic device switches the control state of the endoscope.

Generally, after the automatic stay state is switched, the electronic device may output prompt information, such as output in a voice or text manner, to prompt a user (doctor) to find an abnormal condition, and then execute a corresponding operation according to an operation instruction input by the user, so as to implement a manual operation manner. Or after the endoscope enters the automatic stopping state, corresponding operations can be executed according to preset control instructions such as posture adjustment and image shooting in the automatic stopping state, so that manual control of a user can be waited.

As a preferred embodiment, after the manipulation state of the endoscope is switched from the automatic movement state to the automatic stay state in step S50, the electronic device may further perform the following steps S50 to S90:

and S50, when the duration of the endoscope in the automatic stopping state reaches a specified duration, the electronic equipment controls the endoscope to acquire second image data.

During the moving process of the endoscope, the endoscope may contact the intestinal wall to cause peristaltic stress reaction of the intestinal wall, so that the movement of the biological tissue located on the intestinal wall is accelerated, and it is understood that the moving phenomenon of the biological tissue on the intestinal wall may cause noise (which may be a blur, a ghost image, and the like) to be present in the image data acquired by the endoscope in real time. Therefore, in step S50, by continuing the automatic stopping state for a certain duration, the image data can be acquired after the environment in the cavity is in the relatively static state, so as to improve the definition and accuracy of the acquired image data. The specific value of the specified time period may be preset by a developer, for example, set to 5 seconds, 10 seconds, or 20 seconds. The specific implementation of the image data acquisition may be to capture a plurality of groups of images at the observation node to obtain an image data set, and select an image with clear image contour and uniform image frame brightness in the image data set as the second image data.

The controlling the endoscope to acquire the second image data specifically may include:

firstly, comparing the current three-dimensional space map with a preset three-dimensional space map to obtain a region where distinguishing data is located, wherein the region is considered to be a region where abnormal biological tissues are located, namely an obstacle region. The obstacle region refers to an approximate positional range of an abnormal biological tissue. Then, the posture of the endoscope is adjusted according to the obstacle region, for example, the endoscope is controlled to bend upward/downward so that the image pickup direction of an image sensor such as a CCD, a CMOS, or a CIS provided at the distal end of the endoscope is directed toward the obstacle region, and finally the endoscope is controlled to acquire second image data in the adjusted posture.

It is understood that the abnormal biological tissue is a diseased biological tissue that is unknown from the CT image or the MRI image, such as a biological tissue that produces an inflammatory phenomenon, a biological tissue that produces a congestive phenomenon, a biological tissue that produces a physiological polyp (non-diseased), and the like.

And S60, the electronic equipment constructs a target three-dimensional space map according to the second image data.

Since the second image data is obtained after waiting for the environment in the lumen to be in a relatively stationary state or adjusting the posture of the endoscope in accordance with the obstacle region. In contrast, the accuracy of the second image data is higher than that of the first image data. Therefore, a more accurate three-dimensional space map of the target can be constructed from the second image data.

In some possible embodiments, after acquiring the second image data, the first image data and the second image data may be fused to obtain target image data, and then a further accurate target three-dimensional space map is constructed preferably according to the target image data.

And S70, the electronic equipment compares the target three-dimensional space map with a preset three-dimensional space map to obtain the position and the shape of the obstacle.

In the step, the target three-dimensional space map can be compared with a preset three-dimensional space map to obtain a distinguished image area, namely a distinguished map area, and then a plurality of positions to be selected are identified according to the distinguished map area; for accurate positioning, discrimination may be performed in combination with the above-determined obstacle area, so as to determine a candidate position located in the obstacle area as an obstacle position. Furthermore, image recognition can be carried out on the specified neighborhood taking the obstacle position as the center in the distinguishing map domain to obtain the obstacle shape, so that the information of the shape, the size, the severity and the like of the obstacle can be known.

And S80, planning an obstacle avoidance channel by the electronic equipment based on the target three-dimensional space diagram according to the position and the shape of the obstacle.

In the application scenario of the present invention, the endoscope needs to traverse through the various observation nodes and finally move to the endpoint. If the abnormal biological tissues (namely the obstacles) are determined to exist nearby in a certain observation node, the positions and the shapes of the obstacles can be further intelligently judged by the electronic equipment besides guiding the endoscope to move through the artificial intervention, and an obstacle avoidance channel is planned. Judging whether the space size of the obstacle avoidance channel is enough for the endoscope to pass through without obstacles and moving to the next observation node; if not, the electronic equipment outputs prompt information for prompting the user to manually control, and executes corresponding operation according to an operation instruction input by the user; if yes, go to step S90.

And S90, if the space size of the obstacle avoidance channel is enough for the endoscope to pass through without obstacles and move to the next observation node, the electronic equipment switches the control state of the endoscope from the automatic stop state to the automatic moving state.

If the target three-dimensional space map is enough, the electronic equipment can store the target three-dimensional space map and the position of the obstacle in a correlated mode, then control the control state of the endoscope to be switched from the automatic stopping state to the automatic moving state, so that the endoscope can further move to the next observation node, and the operation of the steps S10 to S90 is repeatedly executed.

Therefore, after each observation node on the preset planning path is traversed, the actual cavity environment can be identified and analyzed section by section, so that the flexible switching between the automatic moving state and the automatic staying state of the endoscope is controlled.

As shown in fig. 2, the embodiment of the present invention discloses an endoscope manipulation state automatic switching device, which comprises a first acquisition unit 201, a first construction unit 202, a calculation unit 203, and a switching unit 204, wherein,

a first collecting unit 201, configured to control the endoscope to collect and obtain first image data at an observation node when the endoscope is in an automatic moving state and moves to any observation node on a preset planned path;

a first constructing unit 202, configured to construct a current three-dimensional space map according to the first image data;

a calculating unit 203, configured to calculate a matching degree between the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node;

a switching unit 204, configured to switch the endoscope operation state from the automatic moving state to the automatic staying state when the matching degree calculated by the calculating unit 203 is smaller than the matching threshold.

As an alternative embodiment, the endoscope manipulation state automatic switching device may further include:

a second acquisition unit 205, configured to control the endoscope to acquire second image data after the switching unit 204 switches the manipulation state of the endoscope from the automatic moving state to the automatic staying state, and when a duration of the endoscope in the automatic staying state reaches a specified duration;

a second construction unit 206, configured to construct a target three-dimensional space map according to the second image data;

the identification unit 207 is used for comparing the target three-dimensional space diagram with a preset three-dimensional space diagram to obtain the position and the shape of the obstacle;

the obstacle avoidance planning unit 208 is used for planning an obstacle avoidance channel based on the target three-dimensional space map according to the position and the shape of the obstacle;

the switching unit 204 is further configured to switch the control state of the endoscope from the automatic stay state to the automatic moving state when the space size of the obstacle avoidance channel planned by the obstacle avoidance planning unit 208 is enough for the endoscope to pass through without an obstacle and move to the next observation node.

Optionally, the identifying unit 207 is specifically configured to compare the target three-dimensional space map with a preset three-dimensional space map to obtain a difference map domain; identifying a plurality of positions to be selected according to the distinguishing graph field; determining a position to be selected in the obstacle area as an obstacle position; and carrying out image recognition on the specified neighborhood taking the obstacle position as the center to obtain the obstacle form.

Optionally, the second collecting unit 205 may include the following sub-units not shown in the figure:

the comparison subunit is used for comparing the current three-dimensional space map with a preset three-dimensional space map to obtain an obstacle area when the duration of the endoscope in the automatic stay state reaches a specified duration;

an adjustment subunit for adjusting the posture of the endoscope according to the obstacle region;

As an optional implementation manner, the endoscope manipulation state automatic switching device may further include a fusion unit, not shown, for fusing the first image data and the second image data after the second acquisition unit controls the endoscope to acquire the second image data, so as to acquire target image data;

correspondingly, the second constructing unit 206 is specifically configured to construct the target three-dimensional space map according to the target image data.

As shown in fig. 3, an embodiment of the present invention discloses an electronic device, which includes a memory 301 storing executable program codes and a processor 302 coupled to the memory 301;

the processor 302 calls the executable program code stored in the memory 301 to execute the endoscope operation state automatic switching method described in the above embodiments.

The embodiment of the invention also discloses a computer readable storage medium which stores a computer program, wherein the computer program enables a computer to execute the endoscope control state automatic switching method described in the embodiments.

The above embodiments are provided to illustrate, reproduce and deduce the technical solutions of the present invention, and to fully describe the technical solutions, the objects and the effects of the present invention, so as to make the public more thoroughly and comprehensively understand the disclosure of the present invention, and not to limit the protection scope of the present invention.

The above examples are not intended to be exhaustive of the invention and there may be many other embodiments not listed. Any alterations and modifications without departing from the spirit of the invention are within the scope of the invention.

Claims

1. An endoscope control state automatic switching method is characterized by comprising the following steps:

2. The endoscope manipulation state automatic switching method according to claim 1, wherein after switching the manipulation state of the endoscope from the automatic moving state to an automatic staying state, the method further comprises:

when the duration of the endoscope in the automatic stay state reaches a specified duration, controlling the endoscope to acquire second image data;

planning an obstacle avoidance channel based on the target three-dimensional space map according to the obstacle position and the obstacle form;

and if the space size of the obstacle avoidance channel is enough for the endoscope to pass through without obstacles and move to the next observation node, switching the control state of the endoscope from the automatic stopping state to the automatic moving state.

3. The endoscope manipulation state automatic switching method according to claim 2, wherein controlling the endoscope to acquire second image data comprises:

comparing the current three-dimensional space map with the preset three-dimensional space map to obtain an obstacle region;

adjusting the posture of the endoscope according to the obstacle area;

4. The method for automatically switching the operation state of an endoscope according to claim 3, wherein the step of comparing the target three-dimensional space diagram with the preset three-dimensional space diagram to obtain the position and shape of the obstacle comprises:

5. The endoscope manipulation state automatic switching method according to any one of claims 2 to 4, wherein after controlling the endoscope to acquire the second image data, the method further comprises:

and constructing a target three-dimensional space map according to the second image data, wherein the construction comprises the following steps:

6. An endoscope operation state automatic switching device, comprising:

the calculation unit is used for calculating the matching degree of the current three-dimensional space diagram and a preset three-dimensional space diagram corresponding to the observation node;

7. The endoscope manipulation state automatic switching device according to claim 6, further comprising:

8. The endoscope manipulation state automatic switching device according to claim 7, wherein the second acquisition unit comprises:

9. An electronic device comprising a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory for executing the endoscope manipulation state automatic switching method according to any one of claims 1 to 5.

10. A computer-readable storage medium characterized in that the computer-readable storage medium stores a computer program, wherein the computer program causes a computer to execute the endoscope manipulation state automatic switching method according to any one of claims 1 to 5.