CN117690134B - Method and device for assisting in marking target position of electrotome in ESD operation - Google Patents
Method and device for assisting in marking target position of electrotome in ESD operation Download PDFInfo
- Publication number
- CN117690134B CN117690134B CN202410149126.9A CN202410149126A CN117690134B CN 117690134 B CN117690134 B CN 117690134B CN 202410149126 A CN202410149126 A CN 202410149126A CN 117690134 B CN117690134 B CN 117690134B
- Authority
- CN
- China
- Prior art keywords
- point
- marking
- boundary
- points
- distance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 230000003902 lesion Effects 0.000 claims abstract description 20
- 239000003550 marker Substances 0.000 claims description 23
- 230000006870 function Effects 0.000 claims description 18
- 230000011218 segmentation Effects 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 5
- 238000001356 surgical procedure Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 210000000436 anus Anatomy 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 claims description 2
- 230000000740 bleeding effect Effects 0.000 abstract description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000005520 cutting process Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 230000001788 irregular Effects 0.000 abstract description 2
- 239000007788 liquid Substances 0.000 abstract description 2
- 238000012323 Endoscopic submucosal dissection Methods 0.000 description 29
- 238000010586 diagram Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 206010028980 Neoplasm Diseases 0.000 description 4
- 230000003247 decreasing effect Effects 0.000 description 3
- XKRFYHLGVUSROY-UHFFFAOYSA-N Argon Chemical compound [Ar] XKRFYHLGVUSROY-UHFFFAOYSA-N 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000015271 coagulation Effects 0.000 description 2
- 238000005345 coagulation Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 210000004877 mucosa Anatomy 0.000 description 2
- 208000000461 Esophageal Neoplasms Diseases 0.000 description 1
- 206010030155 Oesophageal carcinoma Diseases 0.000 description 1
- 229910052786 argon Inorganic materials 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 210000001072 colon Anatomy 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 201000004101 esophageal cancer Diseases 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 230000002439 hemostatic effect Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 201000011591 microinvasive gastric cancer Diseases 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001575 pathological effect Effects 0.000 description 1
- 230000007170 pathology Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of medical image processing, and particularly relates to a marking auxiliary method and device for an electric knife target position in an ESD operation. The beneficial effects of the invention include: the pre-incision path is more definite, especially for flat lesions with irregular boundaries; ensuring negative lateral cutting edges of the specimens after the operation; the bleeding or liquid in the operation affects the observation of the lesion edge, the effect of observing the mark point is better than that of directly observing the edge, and the pre-incision efficiency is improved.
Description
Technical Field
The invention belongs to the technical field of medical image processing, and particularly relates to an auxiliary method and device for marking a target position of an electrotome in an ESD operation.
Background
Endoscopic Submucosal Dissection (ESD) refers to a minimally invasive technique of completely dissecting diseased mucosa from submucosal layer under an endoscope, was first initiated in japan at the end of the 90 th century and applied to clinic, and is a selective diagnostic or radical operation. The main purpose of ESD is to diagnose and treat early digestive tract tumor, which has the advantage of one-time complete excision of superficial lesions with certain area, but has high technical requirement and great difficulty. Is suitable for diagnosis and treatment of early esophageal cancer, early gastric cancer, interstitial tumor and colon early tumor.
The surgical procedure for ESD is specifically as follows: the operation adopts a supine position, stains the lesion, performs endoscopic ultrasonic examination to determine the range and depth of the lesion, performs electric coagulation marking at a position 5mm outside the edge of the lesion by using a needle-shaped electric knife or an argon knife (APC for short), then performs multi-point submucosal injection outside the edge marking point of the lesion until the lesion is obviously raised, cuts the mucosa along the position 5mm outside the edge marking point of the lesion by using the electric knife, peels off the submucosal layer of the lesion, and then stops bleeding by using an electric coagulation or hemostatic clamp, determines no active bleeding, and sends the taken specimen to the pathological examination in time.
However, in the prior art, when the electric knife is used for marking the target position in the ESD operation, the situation of low accuracy often occurs, and a marking auxiliary scheme for the target position of the electric knife in the ESD operation is needed.
Disclosure of Invention
According to a first aspect of the present invention, the present invention claims a method for assisting in marking a target position of an electrotome in an ESD operation, comprising:
acquiring a marked endoscope image of an ESD operation, inputting the marked endoscope image into a focus recognition model, and predicting a focus boundary of the marked endoscope image;
performing edge-enlarging treatment on the focus boundary of the marked endoscopic image, and inputting the focus edge-enlarging region after edge enlarging into a boundary identification model;
the boundary recognition model predicts a marked boundary of an electrotome in an ESD operation based on the focus edge-enlarging region, and draws the marked boundary into a marked endoscopic image;
and according to the marking boundary, obtaining the perimeter of the marking boundary and the number of marking points through a function arcLength in a computer vision open source operation library OpenCV, and predicting and outputting the marking points of the electric knife.
Further, the step of obtaining a marking endoscope image of the ESD operation, inputting the marking endoscope image into a focus recognition model, and predicting a focus boundary of the marking endoscope image, further comprises:
the focus recognition model comprises a backbone network, a neg, a box_head, a mask branch and a mask head;
the network structure used by the main network is ResNet-50, the convolution basic structure in the main network is expressed as [ convolution kernel size, output channel number ]. Times of circulation, and the convolution in the main network is provided with a relu activation function;
the Neck part adopts an FPN characteristic pyramid structure;
the box_head completes classification of candidate frames and detection of target positions from the feature map, and comprises two branches, wherein each branch is a convolution of 1*1, and an output channel is 256, so that classification prediction and position prediction of the marker endoscope image are respectively completed;
inputting a P3 feature map of the FPN into the mask branch, and performing 4 continuous convolution layer operations of 3x3 to obtain an output Fmask;
performing concat connection on the Fmask and the relative coordinate graph to obtain Rmask;
sequentially passing the upsampled Rmask through a filter combination theta to obtain a binary image, and representing mask segmentation results of corresponding examples;
and dividing the marked endoscopic image into N groups of mask heads to obtain N example division results.
Further, the edge-enlarging processing is performed on the focus boundary of the marking endoscope image, and the focus edge-enlarging region after edge enlarging is input into a boundary recognition model, and the method further comprises the following steps:
acquiring the size of a focus boundary of the marking endoscope image, and extracting the length and the width of the size of the focus boundary;
setting edge expansion percentage, and carrying out edge expansion treatment on an upper boundary, a lower boundary, a left boundary and a right boundary on the focus boundary according to the edge expansion percentage to obtain a candidate focus edge expansion region;
and scaling the candidate focus edge-enlarging region to obtain a focus edge-enlarging region, and inputting the focus edge-enlarging region into a boundary recognition model.
Further, according to the marking boundary, the perimeter of the marking boundary and the number of marking points are obtained through a function arcLength in a computer vision open source computing library OpenCV, and the method predicts and outputs the marking points of the electric knife further comprises the following steps:
obtaining the perimeter of the marked boundary and the number of marked points through a function arcLength in a computer vision open source operation library OpenCV;
when the anal side points are dense and the mouth side points are thin, drawing a first point from the uppermost part of the marked boundary, and then gradually lengthening the interval between each point;
when the anus side points are thin and the mouth side points are dense, drawing a first point from the lowest part of the marked boundary, and then gradually lengthening the interval between each point;
constructing a mark point array according to the number of the mark points, and sequentially calculating Euclidean distances between the mark points and the next point from the first point of the mark point array, and superposing until all points are traversed;
acquiring an upper boundary marking point and a lower boundary marking point from the marking boundary, and calculating the left half perimeter and the right half perimeter of the marking boundary;
obtaining middle boundary marking points in the left half perimeter and the right half perimeter, calculating the distance between the first point in the left half perimeter and the right half perimeter and the lower boundary marking point according to an arithmetic sum formula, and then sequentially calculating the distance between each remaining boundary marking point and the lower boundary marking point;
all boundary marker points were found again using arcLength and coordinate traversal as electrotome marker points.
Further, the obtaining the perimeter of the marking boundary and the number of marking points through the function arcLength in the open source computing library OpenCV of computer vision further includes:
converting the marked boundary of the electrotome boundary recognition contour map into polygon orderPoint sets, resulting in a point set array of M ({ (x) 1 ,y 1 ), (x 2 , y 2 ), ... (x n , y n )}）；
Calculating the total perimeter surrounded by each point in the point set array M by arcLength;
calculating the total number of required mark points, wherein x=floorodd (sqrt (circumference)/3) +2, wherein sqrt represents an evolution operation, floorOdd represents a rounding down after division, and if the rounding result is an odd number, the whole result is +1;
according to the coordinates in the point set array M, two points with maximum and minimum longitudinal coordinates Y are counted and used as the uppermost point and the lowermost point of the marking points of the outline;
marking the point set as R, wherein the R has two points in total;
the uppermost point to the lowermost point is the left half perimeter S if the ordinate is monotonically decreasing l On the contrary, the right half circumference S r 。
Further, the step of constructing a mark point array according to the number of the mark points, sequentially calculating euclidean distances between the mark points and the next point from the first point of the mark point array, and overlapping until all points are traversed, and further comprises the steps of:
calculating the required mark points of each half perimeter according to the calculated required mark point number x, and marking as n, n= (x-2)/2,
calculating the marking point of the left half perimeter to obtain a marking point increment interval e= (S) l /(n+1)) = [ S ] 0.1, and then the distance from the first point on the left to the lowest point is a1= [ S ] according to the improved formula of arithmetic summation l - n(n+1) * e/2]/(n+1) to give a 1 ；
According to a n+1 = a 1 +n.d, the distance between each point and the previous point is calculated, and the distance between the first point of the left half circle and the lowest point is a 1 The second point of the left half cycle is a distance a from the first point of the left half cycle 2 The third point of the left half circle is a distance a from the second point of the left half circle 3 Similarly, the density of dots follows the density of dotsIs a rule of (2).
Further, the method further comprises the following steps:
after each point distance of the left half cycle is obtained, the first point L 1 Distance from the lowest point is a 1 ，L 2 Distance from the lowest point is a 1 +a 2 ，L 3 Distance from the lowest point is a 1 +a 2 +a 3 And similarly, sequentially solving the distance from each remaining boundary mark point to the lower boundary mark point;
based on the point set array M, two points Y therein min And Y is equal to max Having been added to the set of marker points R, then Y is located in M min To Y max Right half-cycle continuous point set M r Set of points M contiguous with the left half cycle l ；
Sequentially calculating the distance from each point in the point set Ml of the left half cycle to the lowest point by using arcLength with the lowest point as a starting point;
traversing the dotting distance array { a } 1 ，a 1 +a 2 ，a 1 +a 2 +a 3 ，...，a 1 +a 2 +..+a n And a distance array { k } for each point in the point set Ml of the left half-cycle from the lowest point 1 , k 2 , ..., k w }；
Finding the left half circle and the right half circle to obtain an original point set M of the contour, and easily distinguishing the points of the left half circle and the right half circle after finding the upper vertex and the lower vertex;
the right half circle mark point is obtained in the same manner.
According to a second aspect of the present invention, the present invention claims a marking aid for an electrotome target site in an ESD operation, comprising:
one or more processors;
and a memory having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the method of assisting in marking a target location of an electrotome in an ESD operation.
The invention belongs to the technical field of medical image processing, and particularly relates to a marking auxiliary method and device for an electric knife target position in an ESD operation. The beneficial effects of the invention include: the pre-incision path is more definite, especially for flat lesions with irregular boundaries; ensuring negative lateral cutting edges of the specimens after the operation; the bleeding or liquid in the operation affects the observation of the lesion edge, the effect of observing the mark point is better than that of directly observing the edge, and the pre-incision efficiency is improved.
Drawings
FIG. 1 is a flowchart of an exemplary method for assisting in marking a target position of an electrotome during an ESD operation;
FIG. 2 is a schematic diagram of a marking boundary of an auxiliary method for marking a target position of an electrotome in an ESD operation according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a marker point of an electrotome for an assisted method for marking a target position of the electrotome during an ESD operation according to an embodiment of the present invention;
FIG. 4 is a schematic view of a first polygon set of outline points of an auxiliary method for marking a target position of an electrotome in an ESD operation according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a second polygon set of outline points of an auxiliary method for marking a target position of an electrotome in an ESD operation according to an embodiment of the present invention;
FIG. 6 is a schematic view of the point set marking results of an auxiliary method for marking the target position of an electrotome in an ESD operation according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an auxiliary device for marking a target position of an electrotome in ESD surgery according to an embodiment of the invention.
Detailed Description
Example segmentation (Instance Segmentation) is an image segmentation technique in computer vision that aims at pixel-level segmentation and classification of each target object in an image. It not only can segment different objects in an image but also can identify their boundaries and shape. Thus, example segmentation has a very important role in some application scenarios, such as vehicle and pedestrian detection in automatic driving, tumor segmentation in medical image analysis, part detection in industrial production, etc. Example segmentation methods are typically based on deep-learning Convolutional Neural Networks (CNNs) plus segmentation heads that enable end-to-end learning and processing of the image to achieve semantic understanding and interpretation of each pixel in the image.
According to a first embodiment of the present invention, referring to fig. 1, the present invention claims a method for assisting marking of a target position of an electrotome in ESD surgery, comprising:
acquiring a marked endoscopic image of the ESD operation, inputting the marked endoscopic image into a focus recognition model, and predicting a focus boundary of the marked endoscopic image;
performing edge-enlarging treatment on the focus boundary of the marked endoscope image, and inputting the focus edge-enlarging region after edge enlarging into a boundary identification model;
the boundary recognition model predicts a marking boundary of an electrotome in the ESD operation based on a focus edge-enlarging region, and draws the marking boundary into a marking endoscope image;
according to the marking boundary, the perimeter of the marking boundary and the number of marking points are obtained through a function arcLength in a computer vision open source operation library OpenCV, and the marking points of the electric knife are predicted and output.
Further, acquiring a marking endoscope image of the ESD operation, inputting the marking endoscope image into a focus recognition model, predicting a focus boundary of the marking endoscope image, and further comprising:
the focus recognition model comprises a backbone network, a neg, a box_head, a mask branch and a mask head;
the network structure used by the main network is ResNet-50, the convolution basic structure in the main network is represented as [ convolution kernel size, output channel number ]. Times of circulation, and convolution in the main network is provided with a relu activation function;
the Neck part adopts an FPN characteristic pyramid structure;
the box_head completes classification of candidate frames and detection of target positions from the feature map, and comprises two branches, wherein each branch is a convolution of 1*1, and an output channel is 256, so that classification prediction and position prediction of the marked endoscopic image are respectively completed;
inputting a feature map of the FPN into the mask branch, and performing 4 continuous convolution layer operations of 3x3 to obtain an output Fmask;
performing concat connection on Fmask and the relative coordinate graph to obtain Rmask;
sequentially passing the upsampled Rmask through a filter combination theta to obtain a binary image, and representing mask segmentation results of corresponding examples;
and marking the segmentation of the endoscope image into N groups of mask heads to obtain N example segmentation results.
Further, edge-enlarging processing is performed on the focus boundary of the marking endoscope image, and the focus edge-enlarging region after edge enlarging is input into a boundary recognition model, and the method further comprises the following steps:
acquiring the size of a focus boundary of a marked endoscope image, and extracting the length and the width of the size of the focus boundary;
setting edge expansion percentage, and carrying out edge expansion treatment on the upper boundary, the lower boundary, the left boundary and the right boundary of the focus boundary according to the edge expansion percentage to obtain candidate focus edge expansion areas;
and (3) scaling the candidate focus edge-enlarging region to obtain a focus edge-enlarging region, and inputting the focus edge-enlarging region into a boundary recognition model.
Wherein in this embodiment the strategy for acquisition is 25% each in length and width. Taking a bounding box 200x300 (width x height) size lesion as an example, the left side will be expanded to 200x 25% = 50 pixels, the right side is again 50 pixels. Both the upper and lower edges will be enlarged to 300x 25% = 75 pixels, with the final size after the expansion being 300x450. The lesions after edge expansion are then uniformly scaled to 224x224 before entering the boundary recognition model.
The boundary recognition model also functions as an example segmentation, and the specific model structure is similar to the focus recognition model, and the difference is mainly that the size of the input image is reduced, and the calculation amount is also reduced due to the reduction of the input image. Because of the two-level instance segmentation, the boundary recognition model is more focused on the prediction of the marker boundary without interference from other background information. Training data as used herein is the line of points where the doctor actually makes the electrotome marks. Since the prediction of the boundary recognition model is based on 224x224, the marked boundary needs to be restored to the level of the final original endoscopic image, and the result is shown in fig. 2.
Further, according to the marking boundary, the perimeter of the marking boundary and the number of marking points are obtained through a function arcLength in a computer vision open source operation library OpenCV, and the electric knife marking points are predicted and output, and the method further comprises the following steps:
obtaining the perimeter of the marked boundary and the number of marked points through a function arcLength in a computer vision open source operation library OpenCV;
when the anal side points are dense and the mouth side points are thin, drawing a first point from the uppermost part of the marked boundary, and then gradually lengthening the interval between each point;
when the anus side points are thin and the mouth side points are dense, drawing a first point from the lowest part of the marked boundary, and then gradually lengthening the interval between each point;
constructing a mark point array according to the number of the mark points, sequentially calculating Euclidean distances between the mark points and the next point from the first point of the mark point array, and superposing until all points are traversed;
acquiring an upper boundary marking point and a lower boundary marking point from the marking boundary, and calculating the left half perimeter and the right half perimeter of the marking boundary;
obtaining middle boundary marking points in the left half perimeter and the right half perimeter, calculating the distance between the first point in the left half perimeter and the right half perimeter and the lower boundary marking point according to an arithmetic sum formula, and then sequentially calculating the distance between each remaining boundary marking point and the lower boundary marking point;
all boundary marker points were found again using arcLength and coordinate traversal as electrotome marker points.
In this embodiment, the marking edges are used to perform the pointing operation by means of the edges, which are calculated at the level of the original endoscopic image. It is clear that the orientation of the specimen needs to be distinguished because the ESD cut lesions need to be sent to a pathology department for further examination. For this, we propose a non-uniform marking point for distinguishing the oral side from the anal side, and refer to fig. 3, which is a schematic diagram of a specific electrotome marking point.
Further, obtaining the perimeter of the marking boundary and the number of marking points through a function arcLength in the computer vision open source computing library OpenCV, and further includes:
converting the marked boundary of the electrotome boundary recognition contour map into a polygon ordered point set to obtain a point set array M ({ (x) 1 ,y 1 ), (x 2 , y 2 ), ... (x n , y n )}）；
Calculating the total perimeter surrounded by each point in the point set array M by arcLength;
calculating the total number of required mark points, wherein x=floorodd (sqrt (circumference)/3) +2, wherein sqrt represents an evolution operation, floorOdd represents a rounding down after division, and if the rounding result is an odd number, the whole result is +1;
according to coordinates in the point set array M, two points with maximum and minimum ordinate Y are counted and used as the uppermost point and the lowermost point of the marked points of the outline;
marking the point set as R, wherein the R has two points in total;
the uppermost point to the lowermost point is the left half perimeter S if the ordinate is monotonically decreasing l On the contrary, the right half circumference S r 。
Further, a mark point array is constructed according to the number of the mark points, euclidean distance between the mark points and the next point is calculated from the first point of the mark point array, and the method is overlapped until all points are traversed, and further comprises the following steps:
calculating the required mark points of each half perimeter according to the calculated required mark point number x, and marking as n, n= (x-2)/2
Calculating the mark point of the left half perimeter to obtain the increment interval e= (S) l /(n+1)) = [ S ] 0.1, and then the distance from the first point on the left to the lowest point is a1= [ S ] according to the improved formula of arithmetic summation l - n(n+1) * e/2]/(n+1) to give a 1 ；
According to a n+1 = a 1 +n.d, the distance between each point and the previous point is calculated, and the distance between the first point of the left half circle and the lowest point is a 1 The second point of the left half cycle is a distance a from the first point of the left half cycle 2 The third point of the left half circle is a distance a from the second point of the left half circle 3 Similarly, the density of dots follows the rule of sparse under-density and sparse over-density.
Further, the method further comprises the following steps:
after each point distance of the left half cycle is obtained, the first point L 1 Distance from the lowest point is a 1 ，L 2 Distance from the lowest point is a 1 +a 2 ，L 3 Distance from the lowest point is a 1 +a 2 +a 3 And similarly, sequentially solving the distance from each remaining boundary marking point to the lower boundary marking point;
based on the point set array M, two points Y therein min And Y is equal to max Having been added to the set of marker points R, then Y is located in M min To Y max Right half-cycle continuous point set M r Set of points M contiguous with the left half cycle l ；
Sequentially calculating the distance from each point in the point set Ml of the left half cycle to the lowest point by using arcLength with the lowest point as a starting point;
traversing the dotting distance array { a } 1 ，a 1 +a 2 ，a 1 +a 2 +a 3 ，...，a 1 +a 2 +..+a n And a distance array { k } for each point in the point set Ml of the left half-cycle from the lowest point 1 , k 2 , ..., k w }；
Finding the left half circle and the right half circle to obtain an original point set M of the contour, and easily distinguishing the points of the left half circle and the right half circle after finding the upper vertex and the lower vertex;
the right half circle mark point is obtained in the same manner.
The specific explanation of the point set array and the marker point set for this embodiment is as follows:
first, for the edge of any polygonal object, it can be represented by a number of points, and the set of the points is called a point set array of the polygon, and the point set is two concepts.
Referring to fig. 4, taking this car as an example, the second graph is his detailed outline, the third graph is his polygonal representation, and if the more points in the third graph are concentrated, the closer the third graph is to the second graph, the higher the calculated approximation.
Referring to fig. 5, in this polygonal representation, the left graph is the original outline and the right is the approximate point set representation, so the outline in the right graph passes through 72 points in total, so the first input parameter of the arcLength function is the point set array containing 72 coordinates.
The specific codes of arcLength are as follows:
def arcLength(contour,closed):
length = 0.0
for i in range(len(contour) - 1):
x1, y1 = contour[i][0]
x2, y2 = contour[i+1][0]
distance = ((x2 – x1) ** 2 + (y2 – y1) ** 2) ** 0.5
length +=distance
if the contour is closed, it is necessary to add the distance between the two ending points
if closed and len(contour)>1:
x1, y1 = contour[-1][0]
x2, y2 = contour[0][0]
distance = ((x2 – x1) ** 2 + (y2 – y1) ** 2) ** 0.5
length +=distance
return length
The internal implementation of this function is mainly by calculating the euclidean distance between each adjacent point on the contour and summing these distances. Specifically, for a set of points P1 (x 1, y 1), P2 (x 2, y 2) on a contour, the Euclidean distance d between the points can be calculated by the following formula:
finally, the perimeter or arc length of the contour is obtained, and a second parameter closed of arcLength indicates whether the contour is closed or not, if so, the perimeter of the contour is calculated, and if not, the arc length is calculated.
The step of calculating the marker points is described again:
the contour diagram of the electrotome boundary recognition model is converted into a polygon ordered point set expression form, and the point set is recorded as M ({ (x) 1 ,y 1 ), (x 2 , y 2 ), ... (x n , y n ) }). It should be noted that only a small portion of the points in the set M may become marker points.
Calculating the total circumference of the point set M by using arcLength;
the number of total required marker points, x=floorodd (sqrt (circumference)/3) +2, was calculated. Where sqrt denotes an evolution operation; floorOdd denotes rounding down after division, and if the result after rounding is odd, the whole result+1.
According to the coordinates in the point set M, the two points with the maximum and minimum ordinate Y are counted and used as the uppermost point and the lowermost point of the marked points of the outline. We mark the final result as R, then there are two points in total in R at this time.
The uppermost point to the lowermost point is referred to as the left half perimeter S if the ordinate is constant decreasing l On the contrary, the right half circumference S r . Since the total number of required marker points x has been calculated, each halfThe required mark point for the perimeter is also determined and is denoted as n, n= (x-2)/2.
The marker point for the left half perimeter is calculated. First, the mark point increment interval e= (S) on the left is calculated l /(n+1)) = [ S ] 0.1, and then the distance from the first point on the left to the lowest point is a1= [ S ] according to the improved formula of arithmetic summation l - n(n+1) * e/2]/(n+1) to give a 1 According to a n+1 The distance from each point to the previous point is found. That is, the first point of the left half cycle is a distance from the lowest point 1 The second point of the left half cycle is a distance a from the first point of the left half cycle 2 The third point of the left half circle is a distance a from the second point of the left half circle 3 And so on. Thus, the density of dots presents a rule of sparse lower density and sparse upper density.
The following are exemplified by specific values:
a lesion with a perimeter s=1000 in the original image resolution, with a number of strokes x=12, then the number of strokes n=5 is required per half cycle. Assume that the left half circumference is 400 and the right half circumference is 600. Then we calculate the left half perimeter.
e=(S l / (n+1))*0.1=(400/6)*0.1=6.67；
a 1 =[Sl - n(n+1) * e/2] / (n+1)=(400-5*6*6.67/2)/6=49.99；
Then sequentially calculating the rest 5 point distances, a 2 =56.66、a 3 =63.33、a 4 =70、a 5 =76.67，a 6 =83.34。
After each point distance of the left half cycle is obtained, the first point L is known 1 Distance from the lowest point is a 1 ，L 2 Distance from the lowest point is a 1 +a 2 L3 is a distance a from the lowest point 1 +a 2 +a 3 … the distance from each remaining boundary marker point to the lower boundary marker point is calculated in turn.
Then based on the original ordered set of contour points M, two of which points Y are known min And Y is equal to max Having been added to the set of annotation points R, then Y is easily found in M min To Y max Right half-cycle continuous point set M r Set of points M contiguous with the left half cycle l 。
In the point set of the left half cycle, the distance from each point in the point set Ml of the left half cycle to the lowest point is calculated sequentially using arcLength with the lowest point as the starting point. At this time, the dotting distance array { a } 1 ，a 1 +a 2 ，a 1 +a 2 +a 3 ，...，a 1 +a 2 +..+a n -and a point set M of the left half-cycle l Distance array { k for each point from the lowest point 1 , k 2 , ..., k w It should be noted that the length of the first array is certainly smaller than the length of the second array. Point set M when left half cycle l A certain point in (a) is at a distance from the lowest point which is just larger than a at a certain moment 1 At least one point is assumed to be k 8 Then at least specify L 1 At k 7 And k 8 In the middle, at this time another k 8 This point is added as a marker point to the marker point set R. Similarly, when the point set M is the left half circle l A certain point in (a) is at a distance from the lowest point which is just larger than a at a certain moment 1 +a 2 This point is added to R as the second point of the left half-cycle. Until the cycle traversal of both arrays ends, the marker point for the left half has been found.
An example of marking the left and right half-circle point sets is shown in fig. 6, which is an original point set M of a contour, and the left and right half-circle points can be easily distinguished after finding the upper and lower vertices.
Also taking the example data above as an example, taking the first point of the left half cycle as the case may be, a 1 =49.99, then the left half-cycle point set M is traversed L In which exactly one point is located at a distance from the lowest point just greater than a 1 And adding the other point into the marked point set R until the two-layer cycle traversal is finished, and calculating the marked point of the left half cycle.
In the same way, the right half circle mark point can be obtained by repeating the steps.
According to a second embodiment of the present invention, referring to fig. 7, the present invention claims a marking aid for a target position of an electrotome in an ESD operation, comprising:
one or more processors;
and a memory having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement a method of assisting in marking a target location of an electrotome in an ESD operation.
Those skilled in the art will appreciate that various modifications and improvements can be made to the disclosure. For example, the various devices or components described above may be implemented in hardware, or may be implemented in software, firmware, or a combination of some or all of the three.
A flowchart is used in this disclosure to describe the steps of a method according to an embodiment of the present disclosure. It should be understood that the steps that follow or before do not have to be performed in exact order. Rather, the various steps may be processed in reverse order or simultaneously. Also, other operations may be added to these processes.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the methods described above may be implemented by a computer program to instruct related hardware, and the program may be stored in a computer readable storage medium, such as a read only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps of the above embodiments may be implemented using one or more integrated circuits. Accordingly, each module/unit in the above embodiment may be implemented in the form of hardware, or may be implemented in the form of a software functional module. The present disclosure is not limited to any specific form of combination of hardware and software.
Unless defined otherwise, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The foregoing is illustrative of the present disclosure and is not to be construed as limiting thereof. Although a few exemplary embodiments of this disclosure have been described, those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the claims. It is to be understood that the foregoing is illustrative of the present disclosure and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The disclosure is defined by the claims and their equivalents.
In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
Claims (8)
1. The utility model provides a mark auxiliary method of electrotome target position in ESD operation, is applied to the scope and peels off operation ESD operation, characterized by that includes:
acquiring a marked endoscope image of an ESD operation, inputting the marked endoscope image into a focus recognition model, and predicting a focus boundary of the marked endoscope image;
performing edge-enlarging treatment on the focus boundary of the marked endoscopic image, and inputting the focus edge-enlarging region after edge enlarging into a boundary identification model;
the boundary recognition model predicts a marked boundary of an electrotome in an ESD operation based on the focus edge-enlarging region, and draws the marked boundary into a marked endoscopic image;
and according to the marking boundary, obtaining the perimeter of the marking boundary and the number of marking points through a function arcLength in a computer vision open source operation library OpenCV, and predicting and outputting the marking points of the electric knife.
2. The method of assisting in marking a target position of an electrotome in an ESD operation of claim 1, wherein said acquiring a marking endoscopic image of the ESD operation, inputting said marking endoscopic image into a lesion recognition model, predicting a lesion boundary of said marking endoscopic image, further comprises:
the focus recognition model comprises a backbone network, a neg, a box_head, a mask branch and a mask head;
the network structure used in the backbone network is ResNet-50, and the convolution basic structure in the backbone network is represented as [ convolution kernel size, number of output channels ]] The number of loops, and the convolutions in the backbone network all carry the relu activation function;
the Neck part adopts an FPN characteristic pyramid structure;
box_head completes candidate box classification and target position detection from the feature map, and comprises two branches, wherein each branch is 11, the output channel is 256, and the category prediction and the position prediction of the marking endoscope image are respectively completed;
inputting a P3 feature map of the FPN into the mask branch, and performing 4 continuous convolution layer operations of 3x3 to obtain an output Fmask;
performing concat connection on the Fmask and the relative coordinate graph to obtain Rmask;
sequentially passing the upsampled Rmask through a filter combination theta to obtain a binary image, and representing mask segmentation results of corresponding examples;
and dividing the marked endoscopic image into N groups of mask heads to obtain N example division results.
3. The method for assisting in marking a target position of an electrotome in an ESD operation of claim 1, wherein the edge-enlarging process is performed on a lesion boundary of the marked endoscopic image, and the edge-enlarged lesion edge-enlarging region is input into a boundary recognition model, further comprising:
acquiring the size of a focus boundary of the marking endoscope image, and extracting the length and the width of the size of the focus boundary;
setting edge expansion percentage, and carrying out edge expansion treatment on an upper boundary, a lower boundary, a left boundary and a right boundary on the focus boundary according to the edge expansion percentage to obtain a candidate focus edge expansion region;
and scaling the candidate focus edge-enlarging region to obtain a focus edge-enlarging region, and inputting the focus edge-enlarging region into a boundary recognition model.
4. The method for assisting in marking a target position of an electrotome in an ESD operation according to claim 1, wherein the step of obtaining the perimeter of the marking boundary and the number of marking points by a function arcLength in a computer vision open source calculation library OpenCV, predicting and outputting the marking points of the electrotome, further comprises:
obtaining the perimeter of the marked boundary and the number of marked points through a function arcLength in a computer vision open source operation library OpenCV;
when the anal side points are dense and the mouth side points are thin, drawing a first point from the uppermost part of the marked boundary, and then gradually lengthening the interval between each point;
when the anus side points are thin and the mouth side points are dense, drawing a first point from the lowest part of the marked boundary, and then gradually lengthening the interval between each point;
constructing a mark point array according to the number of the mark points, and sequentially calculating Euclidean distances between the mark points and the next point from the first point of the mark point array, and superposing until all points are traversed;
acquiring an upper boundary marking point and a lower boundary marking point from the marking boundary, and calculating the left half perimeter and the right half perimeter of the marking boundary;
obtaining middle boundary marking points in the left half perimeter and the right half perimeter, calculating the distance between the first point in the left half perimeter and the right half perimeter and the lower boundary marking point according to an arithmetic sum formula, and then sequentially calculating the distance between each remaining boundary marking point and the lower boundary marking point;
all boundary marker points were found again using arcLength and coordinate traversal as electrotome marker points.
5. The method for assisting in marking a target position of an electrotome in ESD surgery according to claim 4, wherein the obtaining the perimeter of the marked boundary and the number of marked points by the function arcLength in the OpenCV of computer vision open source operation library further comprises:
converting the marked boundary of the electrotome boundary recognition contour map into a polygon ordered point set, and obtaining a point set array as M ({ (x) 1 ,y 1 ), (x 2 , y 2 ), ... (x n , y n )}）；
Calculating the total perimeter surrounded by each point in the point set array M by arcLength;
calculating the total number of required mark points, wherein x=floorodd (sqrt (circumference)/3) +2, wherein sqrt represents an evolution operation, floorOdd represents a rounding down after division, and if the rounding result is an odd number, the whole result is +1;
according to the coordinates in the point set array M, two points with maximum and minimum longitudinal coordinates Y are counted and used as the uppermost point and the lowermost point of the marking points of the outline;
marking the point set as R, wherein the R has two points in total;
if the ordinate from the uppermost point to the lowermost point is monotonousDecreasing to the left half circumference S l On the contrary, the right half circumference S r 。
6. The method for assisting in marking a target position of an electrotome in an ESD operation according to claim 5, wherein said constructing an array of marking points according to the number of marking points, sequentially calculating euclidean distances from a first point of said array of marking points to a next point, and superposing until all points are traversed, further comprises:
calculating the required mark points of each half perimeter according to the calculated required mark point number x, and marking as n, n= (x-2)/2
Calculating the marking point of the left half perimeter to obtain a marking point increment interval e= (S) l / (n+1)）0.1, and then according to an improved formula of arithmetic summation, the distance from the first point on the left to the lowest point is a1= [ S ] l - n(n+1) /> e/2]/(n+1) to give a 1 ；
According to a n+1 = a 1 + nd, finding the distance between each point and the previous point, wherein the distance between the first point of the left half cycle and the lowest point is a 1 The second point of the left half cycle is a distance a from the first point of the left half cycle 2 The third point of the left half circle is a distance a from the second point of the left half circle 3 Similarly, the density of dots follows the rule of sparse under-density and sparse over-density.
7. The method for assisting in marking a target position of an electrotome in an ESD operation of claim 6, further comprising:
after each point distance of the left half cycle is obtained, the first point L 1 Distance from the lowest pointIs separated into a 1 ，L 2 Distance from the lowest point is a 1 +a 2 ，L 3 Distance from the lowest point is a 1 +a 2 +a 3 And similarly, sequentially solving the distance from each remaining boundary mark point to the lower boundary mark point;
based on the point set array M, two points Y therein min And Y is equal to max Having been added to the set of marker points R, then Y is located in M min To Y max Right half-cycle continuous point set M r Set of points M contiguous with the left half cycle l ；
Sequentially calculating the distance from each point in the point set Ml of the left half cycle to the lowest point by using arcLength with the lowest point as a starting point;
traversing the dotting distance array { a } 1 ，a 1 +a 2 ，a 1 +a 2 +a 3 ，...，a 1 +a 2 +..+a n And a distance array { k } for each point in the point set Ml of the left half-cycle from the lowest point 1 , k 2 , ..., k w }；
Finding the left half circle and the right half circle to obtain an original point set M of the contour, and easily distinguishing the points of the left half circle and the right half circle after finding the upper vertex and the lower vertex;
the right half circle mark point is obtained in the same manner.
8. A marking aid for a target position of an electrotome in an ESD operation, comprising:
one or more processors;
a memory having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement a method of assisting in marking a target location of an electrotome in an ESD operation according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410149126.9A CN117690134B (en) | 2024-02-02 | 2024-02-02 | Method and device for assisting in marking target position of electrotome in ESD operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410149126.9A CN117690134B (en) | 2024-02-02 | 2024-02-02 | Method and device for assisting in marking target position of electrotome in ESD operation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117690134A CN117690134A (en) | 2024-03-12 |
| CN117690134B true CN117690134B (en) | 2024-04-12 |
Family
ID=90139474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410149126.9A Active CN117690134B (en) | 2024-02-02 | 2024-02-02 | Method and device for assisting in marking target position of electrotome in ESD operation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117690134B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014097103A1 (en) * | 2012-12-17 | 2014-06-26 | Koninklijke Philips N.V. | Segmentation of breast lesions in 3d ultrasound images |
| CN114869456A (en) * | 2022-05-07 | 2022-08-09 | 无锡市人民医院 | Improved high-frequency tool bit and electric knife for preventing intrinsic muscle layer from being burnt by ESD and using method |
| CN116807614A (en) * | 2023-06-08 | 2023-09-29 | 成都博恩思医学机器人有限公司 | Method for continuously displaying focus area of operation in operation |
| CN116965765A (en) * | 2023-08-01 | 2023-10-31 | 西安交通大学医学院第二附属医院 | Early gastric cancer endoscope real-time auxiliary detection system based on target detection algorithm |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110390351B (en) * | 2019-06-24 | 2020-07-24 | 浙江大学 | Epileptic focus three-dimensional automatic positioning system based on deep learning |
-
2024
- 2024-02-02 CN CN202410149126.9A patent/CN117690134B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014097103A1 (en) * | 2012-12-17 | 2014-06-26 | Koninklijke Philips N.V. | Segmentation of breast lesions in 3d ultrasound images |
| CN114869456A (en) * | 2022-05-07 | 2022-08-09 | 无锡市人民医院 | Improved high-frequency tool bit and electric knife for preventing intrinsic muscle layer from being burnt by ESD and using method |
| CN116807614A (en) * | 2023-06-08 | 2023-09-29 | 成都博恩思医学机器人有限公司 | Method for continuously displaying focus area of operation in operation |
| CN116965765A (en) * | 2023-08-01 | 2023-10-31 | 西安交通大学医学院第二附属医院 | Early gastric cancer endoscope real-time auxiliary detection system based on target detection algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117690134A (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Skourt et al. | Lung CT image segmentation using deep neural networks | |
| CN112150524B (en) | Two-dimensional and three-dimensional medical image registration method and system based on deep learning | |
| Häfner et al. | Delaunay triangulation-based pit density estimation for the classification of polyps in high-magnification chromo-colonoscopy | |
| CN109378068B (en) | Automatic evaluation method and system for curative effect of nasopharyngeal carcinoma | |
| CN110265142B (en) | Auxiliary diagnosis system for restoration image of lesion area | |
| CN113436173B (en) | Abdominal multi-organ segmentation modeling and segmentation method and system based on edge perception | |
| Wang et al. | A method of ultrasonic image recognition for thyroid papillary carcinoma based on deep convolution neural network | |
| Zhang et al. | HIFUNet: multi-class segmentation of uterine regions from MR images using global convolutional networks for HIFU surgery planning | |
| CN116196099A (en) | Cardiovascular intervention operation path planning method, system, storage medium and terminal | |
| CN111047608A (en) | Distance-AttU-Net-based end-to-end mammary ultrasound image segmentation method | |
| CN114372951A (en) | Nasopharyngeal carcinoma positioning and segmenting method and system based on image segmentation convolutional neural network | |
| Bodzioch et al. | New approach to gallbladder ultrasonic images analysis and lesions recognition | |
| CN117690134B (en) | Method and device for assisting in marking target position of electrotome in ESD operation | |
| CN111754530A (en) | Prostate ultrasonic image segmentation and classification method | |
| Ham et al. | Improvement of gastroscopy classification performance through image augmentation using a gradient-weighted class activation map | |
| Jia et al. | Improving intraoperative liver registration in image-guided surgery with learning-based reconstruction | |
| Gao et al. | White-light endoscopic colorectal lesion detection based on improved YOLOv5 | |
| CN109816665B (en) | Rapid segmentation method and device for optical coherence tomography image | |
| CN115439473B (en) | Multi-phase occupation classification method based on interactive grouping attention mechanism | |
| CN116205814A (en) | Medical endoscope image enhancement method, system and computer equipment | |
| CN114081625B (en) | Navigation path planning method, system and readable storage medium | |
| CN111126474B (en) | Confocal laser micro-endoscope digestive tract image identification method and system | |
| CN113069137A (en) | Esophagus tumor segmentation method based on DDAUnet | |
| Lonseko et al. | Semi-Supervised Gastrointestinal Lesion Segmentation Using Adversarial Learning | |
| Dhar et al. | Automatic tracing of mandibular canal pathways using deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |