CN114972144A - Method and device for splicing intestinal images of capsule endoscopy, storage medium and equipment - Google Patents
Method and device for splicing intestinal images of capsule endoscopy, storage medium and equipment Download PDFInfo
- Publication number
- CN114972144A CN114972144A CN202210568399.8A CN202210568399A CN114972144A CN 114972144 A CN114972144 A CN 114972144A CN 202210568399 A CN202210568399 A CN 202210568399A CN 114972144 A CN114972144 A CN 114972144A
- Authority
- CN
- China
- Prior art keywords
- image
- intestinal
- rectangular
- images
- intestinal tract
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000968 intestinal effect Effects 0.000 title claims abstract description 116
- 239000002775 capsule Substances 0.000 title claims abstract description 84
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000001839 endoscopy Methods 0.000 title description 2
- 210000001035 gastrointestinal tract Anatomy 0.000 claims abstract description 165
- 230000009466 transformation Effects 0.000 claims abstract description 22
- 230000000007 visual effect Effects 0.000 claims abstract description 11
- 210000000936 intestine Anatomy 0.000 claims description 7
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 6
- 239000012634 fragment Substances 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000001788 irregular Effects 0.000 description 4
- 238000003745 diagnosis Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000008855 peristalsis Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 241000167880 Hirundinidae Species 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003702 image correction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 201000009240 nasopharyngitis Diseases 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30028—Colon; Small intestine
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Endoscopes (AREA)
Abstract
The invention discloses a method, a device, a storage medium and equipment for splicing intestinal tract images of a capsule endoscope. The method comprises the following steps: respectively cutting each original intestinal tract image to obtain a circular intestinal tract image; carrying out rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set; carrying out polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal images; calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, and performing cutting and splicing treatment according to the axial rotation deviation value to form an intestinal spliced image; and acquiring newly added segment images of the intestinal splicing image, and splicing the newly added segment images to form an intestinal panoramic image. The image visual angle deviation caused by lens inclination is reduced and the image angular deviation caused by the rotation of the capsule endoscope is corrected, so that the intestinal panoramic image is more consistent with the actual contour of the intestinal tract.
Description
Technical Field
The invention belongs to the technical field of medical equipment imaging, and particularly relates to a method and a device for splicing intestinal tract images of a capsule endoscope, a computer-readable storage medium and computer equipment.
Background
The capsule endoscope is an intelligent capsule with a built-in miniature camera, the size of the capsule endoscope is close to that of a common cold capsule, and the capsule endoscope is smooth in shell and easy to swallow. After the examinee swallows the capsule endoscope, the capsule endoscope moves forward along with the peristalsis of the digestive tract, the image inside the digestive tract is shot, signals are transmitted to a data recorder worn by the examinee in a wireless mode, and the daily activities of the examinee are not affected in the whole examination process. Finally, the doctor will take down the recorder and download and analyze the data stored in the recorder in the computer to make a diagnosis, and the capsule will be naturally discharged from the body.
Since the capsule endoscope is generally discharged from the body after flowing in the body for eight hours, tens of thousands or more images are taken during this period. If the condition of a patient is diagnosed by directly using tens of thousands of images, the doctor can have visual fatigue when watching the images, and the possibility of missed diagnosis and even misdiagnosis is increased. Thus, researchers in the field have proposed stitching the obtained images to obtain one or several panoramic images for easy recognition and observation.
However, due to irregular movement in different directions in the process of shooting in the capsule endoscope, for example, the capsule endoscope deflects to cause the shooting direction to incline, the capsule endoscope rotates axially to cause the visual angle deviation between the shot pictures, and under the condition, if the shot original images are directly spliced, the obtained panoramic image and the actual outline of the intestinal tract can be greatly distinguished, and subsequent identification and observation are not facilitated.
Disclosure of Invention
(I) technical problems to be solved by the invention
The technical problem solved by the invention is as follows: how to correct the deviation of the shot intestinal tract image caused by irregular movement of the capsule endoscope so that the spliced panoramic image is more consistent with the actual contour of the intestinal tract.
(II) the technical scheme adopted by the invention
A method for stitching intestinal images by using a capsule endoscope comprises the following steps:
respectively cutting each original intestinal tract image shot by a capsule endoscope to obtain a circular intestinal tract image to form a circular intestinal tract image set;
performing rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set, wherein the images with the depth of field areas are in the same shooting visual angle after the rotation correction processing;
performing polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal tract images to form a rectangular intestinal tract image set;
calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, cutting and splicing the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and cutting and splicing each rectangular intestinal image to form an intestinal spliced image set;
acquiring a newly added segment image of each image in the intestinal splicing image set relative to an adjacent previous image in a time sequence to form a newly added segment image set;
and splicing the images in the newly added segment image set according to a time sequence to form an intestinal panoramic image.
Preferably, the method for performing rotation correction processing on the image with the depth of field area in the circular intestinal tract image set comprises:
judging whether each image in the circular intestinal tract image set has a depth of field direction;
and if so, screening the images with the depth of field direction, and rotating the screened images so as to enable the farthest points of the images in the depth of field direction to be at the same angle.
Preferably, the method for calculating the axial rotation deviation value of each current rectangular intestinal tract image in the rectangular intestinal tract image set and the temporally adjacent previous rectangular intestinal tract image comprises:
acquiring a first matching key point set on the current rectangular intestinal tract image and a second matching key point set on the previous rectangular intestinal tract image, wherein each point in the first matching key point set is matched with each point in the second matching key point set one by one;
acquiring a first central point of the first matching key point set and a second central point of the second matching key point set;
and calculating a coordinate difference value of the first central point and the second central point in the direction of the length of the rectangular intestinal image, and taking the coordinate difference value as an axial rotation deviation value, wherein the direction of the length of the rectangular intestinal image is vertical to the direction of the width of the rectangular intestinal image, and the direction of the width of the rectangular intestinal image is parallel to the movement direction of the capsule endoscope.
Preferably, the method for forming the intestinal tract mosaic image by performing cutting mosaic processing on the current rectangular intestinal tract image according to the axial rotation deviation value comprises the following steps:
cutting a rectangular spliced image from a first side of the current rectangular intestinal tract image along the direction of the width, wherein the width of the rectangular spliced image is the same as that of the current rectangular intestinal tract image, and the length of the rectangular spliced image is equal to the axial rotation deviation value;
moving the residual cut image of the current rectangular intestinal tract image along the length direction so as to enable the coordinate values of the first central point and the second central point to be the same in the length direction;
and splicing the rectangular spliced image to a second side of the current rectangular intestinal image to form an intestinal spliced image, wherein the first side is opposite to the second side.
Preferably, the method for obtaining a newly added segment image of each image in the intestinal tract stitched image set relative to a time-sequence adjacent previous intestinal tract stitched image comprises:
selecting a plurality of continuous matching key points from the current intestinal splicing image and the previous rectangular intestinal image adjacent in time sequence respectively to form two matching key point groups;
calculating affine transformation between corresponding points in the two matching keypoint groups;
and aligning and pasting the current intestinal tract spliced image and the previous rectangular intestinal tract image according to affine transformation and matching point positions between corresponding points in the two matching key point groups, and taking the redundant fragment image of the current intestinal tract spliced image relative to the previous rectangular intestinal tract image as a newly added fragment image after aligning and pasting.
Preferably, before acquiring the newly added segment image, the stitching method further includes a step of removing key points with inconsistent order:
matching key points of the current intestinal spliced image and the previous rectangular intestinal image adjacent in time sequence;
sequentially judging whether the sequence of each pair of matched two key points in the image in which the two key points are respectively located along the length direction is the same;
and if the two key points are the same, the two key points are reserved, and if the two key points are not the same, the two key points are removed.
Preferably, the method for sequentially stitching the images in the newly added segment image set according to the time sequence to form the intestinal panoramic image comprises the following steps:
determining the shooting direction of the original intestinal tract image corresponding to each newly added segment image in the newly added segment image set, wherein the shooting direction comprises the advancing direction of the capsule endoscope and the retreating direction of the capsule endoscope;
inserting a segment connecting image between two adjacent newly-added segment images with different shooting directions in the newly-added segment image set to form an updated newly-added segment image set, wherein the segment connecting image is obtained by mirror image transformation of the newly-added segment image with a time sequence in the two newly-added segment images, and a symmetry axis of the mirror image transformation is perpendicular to the direction of the width of the newly-added segment image;
and splicing the images in the updated newly added segment image set in sequence along the width direction of the newly added segment image according to the time sequence to form the intestinal panoramic image.
The application also discloses splicing apparatus of capsule scope intestinal image, splicing apparatus includes:
the preprocessing unit is used for respectively cutting each original intestinal tract image shot by the capsule endoscope to obtain a circular intestinal tract image to form a circular intestinal tract image set;
the rotation correction unit is used for performing rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set, wherein the images with the depth of field areas are in the same shooting visual angle after the rotation correction processing;
the image unfolding unit is used for carrying out polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal images to form a rectangular intestinal image set;
the cutting and splicing unit is used for calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, cutting and splicing the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and cutting and splicing each rectangular intestinal image to form an intestinal spliced image set;
the newly added segment acquisition unit is used for acquiring newly added segment images of each image in the intestinal splicing image set relative to the adjacent previous image in time sequence to form a newly added segment image set;
and the panoramic image splicing unit is used for splicing the images in the newly added segment image set according to the time sequence order to form the intestinal panoramic image.
The application also discloses a computer readable storage medium, the computer readable storage medium stores a splicing program of the capsule endoscopic intestinal tract image, and the splicing program of the capsule endoscopic intestinal tract image is executed by a processor to realize the splicing method of the capsule endoscopic intestinal tract image.
The application also discloses computer equipment, the computer equipment comprises a computer readable storage medium, a processor and a splicing program of the capsule endoscopic intestinal tract images stored in the computer readable storage medium, and the splicing program of the capsule endoscopic intestinal tract images is executed by the processor to realize the splicing method of the capsule endoscopic intestinal tract images.
(III) advantageous effects
Compared with the splicing method, the splicing method and the splicing device for the capsule endoscope intestinal tract images, disclosed by the invention, have the following technical effects:
the images are corrected twice in the front and back, so that the image visual angle deviation caused by lens inclination is reduced, the image angle deviation caused by capsule endoscope rotation is corrected, and the intestinal panoramic image formed by splicing is more consistent with the actual contour of the intestinal. In addition, during actual splicing, the shooting direction of the picture is considered, so that the pasting direction of the newly added segment images is consistent, and the splicing continuity of the panoramic image is facilitated.
Drawings
Fig. 1 is a flowchart of a method for stitching an intestinal image inside a capsule according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a process of generating a rectangular intestinal image by unfolding a corrected image according to a first embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a process of forming an intestinal tract mosaic image by cutting and stitching a rectangular intestinal tract image according to a first embodiment of the present invention;
fig. 4 is a schematic diagram illustrating an acquisition process of an image of a newly added segment according to a first embodiment of the present invention;
fig. 5 is a schematic diagram of a process of splicing images of newly added segments according to a first embodiment of the present invention;
fig. 6 is a schematic block diagram of a splicing device for intestinal images in a capsule endoscope according to a second embodiment of the present invention;
fig. 7 is a schematic diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Before describing in detail the various embodiments of the present application, the technical idea of the present application is first briefly described: in the prior art, due to irregular movement such as self rotation, lens inclination and the like in the capsule endoscope shooting process, if the shot images are directly spliced, the formed intestinal panoramic image is greatly different from the actual contour of the intestinal, so that the subsequent identification and observation are influenced. Therefore, according to the splicing method of the intestinal tract images of the capsule endoscope, firstly, the images with the depth of field areas are subjected to rotation correction processing, the image visual angle deviation caused by lens inclination is reduced, then, after the images are unfolded under polar coordinates, cutting and splicing processing is carried out, the image angular deviation caused by the rotation of the capsule endoscope is corrected, finally, newly added segment images are spliced to form the intestinal tract panoramic images, through two times of image correction, the image deviation caused by irregular movement of the capsule endoscope is reduced to a great extent, and the spliced intestinal tract panoramic images are more matched with the actual contours of intestinal tracts.
Specifically, as shown in fig. 1, the method for stitching an intestinal tract image in a capsule endoscope of the first embodiment includes the following steps:
step S10: respectively cutting each original intestinal tract image shot by a capsule endoscope to obtain a circular intestinal tract image to form a circular intestinal tract image set;
step S20: performing rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set, wherein the images with the depth of field areas are in the same shooting visual angle after the rotation correction processing;
step S30: carrying out polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal tract images to form a rectangular intestinal tract image set;
step S40: calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, cutting and splicing the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and cutting and splicing each rectangular intestinal image to form an intestinal spliced image set;
step S50: acquiring a newly added segment image of each image relative to an adjacent previous image in a time sequence in the intestinal splicing image set to form a newly added segment image set;
step S60: and splicing the images in the newly added segment image set according to a time sequence to form an intestinal panoramic image.
Since the effective shooting part in the original intestinal tract image shot by the capsule endoscope is not necessarily a standard circular area, when the effective shooting part is not a circular area, the largest circular part needs to be cut out, otherwise, a black blank part is mixed in the splicing process, in step S10, a circular intestinal tract image is obtained after each original intestinal tract image is subjected to a cropping process, and a circular intestinal tract image set is formed.
Furthermore, the intestinal structure is abstracted into an ideal cylinder area, when the center of the slice circle is used for shooting and sampling to the right front, the depth of field area of the picture is just overlapped with the pole area, but when the picture is actually shot in the intestinal tract, due to the reasons of inclination, shaking and the like of a capsule endoscope, the shooting direction is often deviated from the central axis of the right front, and the shooting visual angles of all the images are different. Therefore, in step S20, it is determined whether or not each image in the circular intestinal tract image set has a depth direction, and if so, the images having the depth direction are selected, and the selected images are rotated so that the farthest points of the images in the depth direction are at the same angle. When the depth direction does not exist in the image, the image is obtained by shooting the capsule endoscope at a turning position, and the rotation operation processing is not carried out, so that a correction image set is formed.
Further, in step S30, when the polar coordinate conversion is performed and the images in the corrected image set are expanded, as shown in fig. 2In fig. 2, the left side is a corrected image, the right side is a rectangular intestinal tract image, an edge B is formed on the corrected image by cutting the corrected image along a certain angle, and after the edge B is unfolded, the edge B of the left corrected image is changed into two upper and lower edges B of the right rectangular intestinal tract image 1 、B 2 The center point A of the left corrected image becomes the left edge A of the right rectangular intestine image 1 The circumference C of the left-side corrected image becomes the right edge C1 of the right-side rectangular intestinal image, and a rectangular intestinal image set is obtained by processing each corrected image.
In step S40, the method for calculating the axial rotation deviation value of each current rectangular intestine image in the rectangular intestine image set from the time-series adjacent previous rectangular intestine image includes the following steps:
step S401: and acquiring a first matching key point set on the current rectangular intestinal tract image and a second matching key point set on the previous rectangular intestinal tract image, wherein each point in the first matching key point set is matched with each point in the second matching key point set one by one.
Specifically, a current correction image corresponding to a current rectangular intestinal tract image is subjected to key point matching with a previous correction image corresponding to a previous rectangular intestinal tract image to obtain a key point set on the current correction image and a key point set on the previous correction image, then each point in the key point set on the current correction image is subjected to polar coordinate transformation to obtain a first matching key point set on the current rectangular intestinal tract image, and each point in the key point set on the previous correction image is subjected to polar coordinate transformation to obtain a second matching key point set on the previous rectangular intestinal tract image.
Step S402: acquiring a first central point of a first matching key point set and a second central point of a second matching key point set;
the transverse coordinate value of the first central point is the mean value of the transverse coordinate values of all points in the first matching key point set, and the longitudinal coordinate value of the first central point is the mean value of the longitudinal coordinate values of all points in the first matching key point set; the horizontal coordinate value of the second center point is the mean value of the horizontal coordinate values of all the points in the second matching key point set, and the longitudinal coordinate value of the second center point is the mean value of the longitudinal coordinate values of all the points in the second matching key point set. Here, the horizontal direction is the direction of the width of the rectangular intestinal tract image, and the vertical direction is the direction of the length of the rectangular intestinal tract image.
Step S403: and calculating a coordinate difference value of the first central point and the second central point in the direction of the length of the rectangular intestinal tract image, and taking the coordinate difference value as an axial rotation deviation value, wherein the direction of the length of the rectangular intestinal tract image is vertical to the direction of the width of the rectangular intestinal tract image, and the direction of the width of the rectangular intestinal tract image is parallel to the movement direction of the capsule endoscope.
The capsule endoscope rotates axially in the shooting process, so that a certain angle deviation exists between two adjacent images, the deviation is an axial rotation deviation value, and the axial direction is parallel to the movement direction of the capsule endoscope. Corresponding to the rectangular intestinal image, as shown on the left side of the arrow in fig. 3, the current rectangular intestinal image P is obtained 2 And the previous rectangular intestinal tract image P 1 Of each matching key point (P) 1 、P 2 Shown by the dotted arc lines) are connected in a one-to-one correspondence, the connecting lines tend to incline, and when the capsule endoscope does not axially rotate, the connecting lines are horizontally distributed. Therefore, the coordinate difference value of the first central point and the second central point in the direction of the length of the rectangular intestinal tract image is used as the axial rotation deviation value.
Further, in step S40, the method for forming the intestinal tract mosaic image by performing cropping mosaic processing on the current rectangular intestinal tract image according to the axial rotation deviation value includes the following steps:
step S404: cutting out a rectangular spliced image from the first side of the current rectangular intestinal tract image along the direction of the width, wherein the width of the rectangular spliced image is the same as that of the current rectangular intestinal tract image, and the length of the rectangular spliced image is equal to the axial rotation deviation value. As shown on the left side of the arrow in fig. 3, the current rectangular intestinal tract image P 2 The dotted line frame in (1) represents a rectangular stitched image P 21 The solid line box represents the residual image of the current rectangular intestinal tract image after clippingP 22 The first side is the current rectangular intestinal tract image P 2 Is located on the side of the bottom edge.
Step S405: the residual image P of the current rectangular intestinal tract image after being cut 22 And moving along the length direction to ensure that the coordinate values of the first central point and the second central point in the length direction are the same. As shown in the right side of the arrow in fig. 3, the coordinate values of the first central point and the second central point in the length direction are the same, which may be embodied that the horizontal connection line of each matching key point is in a horizontal state.
Step S406: splicing the rectangles into an image P 21 Spliced on the second side of the current rectangular intestinal tract image to form an intestinal tract spliced image M 2 Wherein the first side is opposite the second side. The second side is the side of the top edge of the current rectangular intestinal tract image.
Further, after the intestinal tract stitched image set is acquired, the mismatch key points existing in each intestinal tract stitched image need to be removed. Specifically, the matching key points of the current intestinal tract spliced image and the matching key points of the previous rectangular intestinal tract image are sequenced along the length direction, and the matching key points with inconsistent sequence are mismatching key points. In an ideal state, all the key points on the adjusted current intestinal splicing image and the previous image are correspondingly matched in sequence, and the matching connecting lines are all horizontal straight lines (shown on the right side of an arrow in fig. 3). However, due to the possibility of mismatching, the sequence is different (for example, a pair of matching key point pairs, the 5 th matching key point in the direction of the length of the current intestinal tract spliced image is matched with the 3 rd matching key point on the previous rectangular intestinal tract image, which is regarded as different in sequence), the 5 th matching key point is regarded as a mismatching key point, and the mismatching key point is removed, which is beneficial to obtaining a new segment image in the subsequent step.
In step S50, the method for obtaining a newly added segment image of each image in the intestinal tract stitched image set relative to a time-series adjacent previous image includes: respectively selecting a plurality of continuous matching key points from the current intestinal splicing image and the previous rectangular intestinal image adjacent in time sequence to formTwo matching keypoint groups; calculating affine transformation between corresponding points in the two matching key point groups; and aligning and pasting the current intestinal tract spliced image and the previous rectangular intestinal tract image according to affine transformation and matching point positions between corresponding points in the two matching key point groups, and taking the redundant fragment image of the current intestinal tract spliced image relative to the previous rectangular intestinal tract image as a newly added fragment image after aligning and pasting. The process of alignment and pasting is shown in FIG. 4, wherein the current intestinal tract splicing image M 2 Relative to the previous rectangular intestinal tract image P 1 The redundant segment image on the right side is the newly added segment image N 2 。
For a segment, the intestinal peristalsis causes the segment to be flat when the capsule endoscope advances and the segment to become wrinkled again when the capsule endoscope retreats. The above reasons may cause inconsistency of images taken in the same section of the intestinal tract, and in order to ensure that all valid information can be completely retained, the partial images are also spliced. Therefore, in step S60, the method for stitching the images in the newly added segment image set in time sequence to form the panoramic intestinal tract image includes the following steps:
step S601: and determining the shooting direction of the original intestinal tract image corresponding to each image in the newly added segment image set, wherein the shooting direction comprises the advancing direction of the capsule endoscope and the retreating direction of the capsule endoscope.
The method for judging the shooting direction of the original intestinal tract image comprises the following steps: for the one-to-one correspondence between each image of the original intestinal tract image set and the images of the rectangular intestinal tract image set, when two adjacent rectangular intestinal tract images in time sequence are pasted together according to the alignment of the step S50, if the feature matching point of the previous rectangular intestinal tract image is closer to the right edge of the rectangular intestinal tract image than the feature matching point of the current rectangular intestinal tract image as a whole, the shooting direction of the current rectangular intestinal tract image is the forward direction of the capsule endoscope, otherwise, the shooting direction of the capsule endoscope is the backward direction of the capsule endoscope. It should be noted that the advancing direction of the capsule endoscope is toward the right edge of the rectangular intestinal tract image.
Step S602: inserting a segment connecting image between two adjacent newly-added segment images with different shooting directions in the newly-added segment image set to form an updated newly-added segment image set, wherein the segment connecting image is obtained by mirror image transformation of the newly-added segment image with a time sequence in the two newly-added segment images, and a symmetry axis of the mirror image transformation is perpendicular to the direction of the width of the newly-added segment image;
step S603: and splicing the images in the updated newly added segment image set in sequence along the width direction of the newly added segment image according to the time sequence to form the intestinal panoramic image. As shown in FIG. 5, the newly added segment image (N) on the left side 1 To N 5 ) The new segment image (N) is obtained by shooting when the capsule endoscope advances 7 To N 11 ) Is obtained by shooting when the capsule endoscope moves backwards, N 6 Connecting images for segments, N 6 From N 5 And obtaining mirror image transformation. Because the two images with changed directions are incompatible, a segment connecting image is inserted between the two images with changed shooting directions, so that the transition effect is achieved, and the continuity on the space can be ensured.
According to the splicing method of the intestinal tract images of the capsule endoscope, the images are corrected twice in the front and the back, so that the image visual angle deviation caused by lens inclination is reduced, the image angle deviation caused by rotation of the capsule endoscope is corrected, and the intestinal tract panoramic images formed by splicing are more consistent with the actual contour of an intestinal tract.
As shown in fig. 6, a second embodiment of the present application further discloses a splicing device for intestinal tract images of a capsule endoscope, which includes a preprocessing unit 100, a rotation correction unit 200, an image unfolding unit 300, a cutting and splicing unit 400, a newly added segment obtaining unit 500, and a panoramic image splicing unit 600. The preprocessing unit 100 is configured to respectively perform clipping processing on each original intestinal tract image captured by a capsule endoscope to obtain a circular intestinal tract image, and form a circular intestinal tract image set; the rotation correction unit 200 is configured to perform rotation correction processing on an image with a depth of field area in the circular intestinal tract image set to obtain a corrected image set, where the image with the depth of field area is at the same shooting angle after the rotation correction processing; the image unfolding unit 300 is configured to perform polar coordinate transformation on each image in the corrected image set and unfold the image into a rectangular intestinal tract image to form a rectangular intestinal tract image set; the cutting and splicing unit 400 is configured to calculate an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a time-sequence adjacent previous rectangular intestinal image, perform cutting and splicing processing on the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and perform cutting and splicing processing on each rectangular intestinal image to form an intestinal spliced image set; the newly added segment acquisition unit 500 is configured to acquire a newly added segment image of each image in the intestinal splicing image set relative to a previous image adjacent in time sequence, and form a newly added segment image set; the panoramic image stitching unit 600 is configured to stitch the images in the newly added segment image set according to a time sequence order to form an intestinal panoramic image. The detailed processing procedures of the preprocessing unit 100, the rotation correcting unit 200, the image expanding unit 300, the clipping and splicing unit 400, the newly added segment obtaining unit 500, and the panoramic image splicing unit 600 refer to the description of the first embodiment, and are not described herein again.
The third embodiment of the application further discloses a computer readable storage medium, wherein a splicing program of the capsule endoscopic intestinal tract images is stored in the computer readable storage medium, and the splicing method of the capsule endoscopic intestinal tract images is realized when the splicing program of the capsule endoscopic intestinal tract images is executed by a processor.
In the third embodiment, a computer device is further disclosed, and in the hardware level, as shown in fig. 7, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads a corresponding computer program from the computer-readable storage medium and then runs, forming a request processing apparatus on a logical level. Of course, besides software implementation, the one or more embodiments in this specification do not exclude other implementations, such as logic devices or combinations of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices. The computer readable storage medium 11 stores a stitching program of the capsule endoscopic intestinal tract image, and the stitching program of the capsule endoscopic intestinal tract image is executed by the processor to realize the stitching method of the capsule endoscopic intestinal tract image.
Computer-readable storage media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
Although a few embodiments of the present invention have been shown and described, it would be appreciated by those skilled in the art that changes may be made in this embodiment without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.
Claims (10)
1. A splicing method of intestinal tract images of a capsule endoscope is characterized by comprising the following steps:
respectively cutting each original intestinal tract image shot by a capsule endoscope to obtain a circular intestinal tract image to form a circular intestinal tract image set;
performing rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set, wherein the images with the depth of field areas are in the same shooting visual angle after the rotation correction processing;
performing polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal tract images to form a rectangular intestinal tract image set;
calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, cutting and splicing the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and cutting and splicing each rectangular intestinal image to form an intestinal spliced image set;
acquiring a newly added segment image of each image in the intestinal splicing image set relative to an adjacent previous image in a time sequence to form a newly added segment image set;
and splicing the images in the newly added segment image set according to a time sequence to form an intestinal panoramic image.
2. The method for stitching an intestinal image under capsule according to claim 1, wherein the method for performing rotation correction processing on the image of the depth area in the circular intestinal image set comprises:
judging whether each image in the circular intestinal tract image set has a depth of field direction;
and if so, screening the images with the depth of field direction, and rotating the screened images so as to enable the farthest points of the images in the depth of field direction to be at the same angle.
3. The method for stitching endoscope images according to claim 1, wherein the method for calculating the axial rotation deviation value of each current rectangular intestine image in the set of rectangular intestine images from the temporally adjacent previous rectangular intestine image comprises:
acquiring a first matching key point set on the current rectangular intestinal tract image and a second matching key point set on the previous rectangular intestinal tract image, wherein each point in the first matching key point set is matched with each point in the second matching key point set one by one;
acquiring a first central point of the first matching key point set and a second central point of the second matching key point set;
and calculating a coordinate difference value of the first central point and the second central point in the direction of the length of the rectangular intestinal tract image, and taking the coordinate difference value as an axial rotation deviation value, wherein the direction of the length of the rectangular intestinal tract image is vertical to the direction of the width of the rectangular intestinal tract image, and the direction of the width of the rectangular intestinal tract image is parallel to the movement direction of the capsule endoscope.
4. The method for splicing intestinal tract images in a capsule endoscope according to claim 3, wherein the method for forming the intestinal tract spliced image comprises the following steps of cutting and splicing the current rectangular intestinal tract image according to the axial rotation deviation value:
cutting a rectangular spliced image from a first side of the current rectangular intestinal tract image along the direction of the width, wherein the width of the rectangular spliced image is the same as that of the current rectangular intestinal tract image, and the length of the rectangular spliced image is equal to the axial rotation deviation value;
moving the residual cut image of the current rectangular intestinal tract image along the length direction so as to enable the coordinate values of the first central point and the second central point to be the same in the length direction;
and splicing the rectangular spliced image to a second side of the current rectangular intestinal image to form an intestinal spliced image, wherein the first side is opposite to the second side.
5. The method for stitching endoscope images according to claim 4, wherein the method for obtaining the newly added segment image of each image in the intestinal stitched image set relative to the temporally adjacent previous image comprises:
selecting a plurality of continuous matching key points from the current intestinal splicing image and the previous rectangular intestinal image adjacent in time sequence respectively to form two matching key point groups;
calculating affine transformation between corresponding points in the two matching keypoint groups;
and aligning and pasting the current intestinal tract spliced image and the previous rectangular intestinal tract image according to affine transformation and matching point positions between corresponding points in the two matching key point groups, and taking the redundant fragment image of the current intestinal tract spliced image relative to the previous rectangular intestinal tract image as a newly added fragment image after aligning and pasting.
6. The method for stitching an endoscopic intestinal image within a capsule according to claim 5, wherein before acquiring the images of the newly added segments, the stitching method further comprises the step of removing the key points with inconsistent sequence:
matching key points of the current intestinal spliced image and the previous rectangular intestinal image adjacent in time sequence;
sequentially judging whether the sequence of each pair of matched two key points in the image in which the two key points are respectively located along the length direction is the same;
and if the two key points are the same, the two key points are reserved, and if the two key points are not the same, the two key points are removed.
7. The method for splicing intestinal tract images in a capsule endoscope according to claim 5, wherein the method for splicing the images in the newly-added segment image set sequentially according to a time sequence to form an intestinal tract panoramic image comprises the following steps:
determining the shooting direction of the original intestinal tract image corresponding to each newly added segment image in the newly added segment image set, wherein the shooting direction comprises the advancing direction of the capsule endoscope and the retreating direction of the capsule endoscope;
inserting a segment connecting image between two adjacent newly-added segment images with different shooting directions in the newly-added segment image set to form an updated newly-added segment image set, wherein the segment connecting image is obtained by mirror image transformation of the newly-added segment image with a time sequence in the two newly-added segment images, and a symmetry axis of the mirror image transformation is perpendicular to the direction of the width of the newly-added segment image;
and splicing the images in the updated newly added segment image set in sequence along the width direction of the newly added segment image according to the time sequence to form the intestinal panoramic image.
8. The utility model provides a splicing apparatus of capsule scope intestinal image which characterized in that, splicing apparatus includes:
the preprocessing unit is used for respectively cutting each original intestinal tract image shot by the capsule endoscope to obtain a circular intestinal tract image to form a circular intestinal tract image set;
the rotation correction unit is used for performing rotation correction processing on the images with the depth of field areas in the circular intestinal tract image set to obtain a corrected image set, wherein the images with the depth of field areas are in the same shooting visual angle after the rotation correction processing;
the image unfolding unit is used for carrying out polar coordinate transformation on each image in the corrected image set and unfolding the images into rectangular intestinal images to form a rectangular intestinal image set;
the cutting and splicing unit is used for calculating an axial rotation deviation value of each current rectangular intestinal image in the rectangular intestinal image set and a previous rectangular intestinal image adjacent in time sequence, cutting and splicing the current rectangular intestinal image according to the axial rotation deviation value to form an intestinal spliced image, and cutting and splicing each rectangular intestinal image to form an intestinal spliced image set;
the newly added segment acquisition unit is used for acquiring newly added segment images of each image in the intestinal splicing image set relative to a previous image adjacent in time sequence to form a newly added segment image set;
and the panoramic image splicing unit is used for splicing the images in the newly added segment image set according to the time sequence order to form the intestinal panoramic image.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a stitching program of an intra-capsule intestinal image, which when executed by a processor implements the method of stitching an intra-capsule intestinal image according to any one of claims 1 to 8.
10. A computer device, characterized in that the computer device comprises a computer readable storage medium, a processor and a stitching program of a capsule endoscopic intestinal image stored in the computer readable storage medium, the stitching program of the capsule endoscopic intestinal image being executed by the processor to implement the stitching method of the capsule endoscopic intestinal image according to any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210568399.8A CN114972144B (en) | 2022-05-23 | 2022-05-23 | Method, device, storage medium and equipment for splicing intestinal images of capsule endoscope |
| PCT/CN2023/095593 WO2023226941A1 (en) | 2022-05-23 | 2023-05-22 | Method and apparatus for stitching intestinal images from capsule endoscope, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210568399.8A CN114972144B (en) | 2022-05-23 | 2022-05-23 | Method, device, storage medium and equipment for splicing intestinal images of capsule endoscope |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114972144A true CN114972144A (en) | 2022-08-30 |
| CN114972144B CN114972144B (en) | 2024-02-02 |
Family
ID=82955094
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210568399.8A Active CN114972144B (en) | 2022-05-23 | 2022-05-23 | Method, device, storage medium and equipment for splicing intestinal images of capsule endoscope |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114972144B (en) |
| WO (1) | WO2023226941A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023226941A1 (en) * | 2022-05-23 | 2023-11-30 | 江苏势通生物科技有限公司 | Method and apparatus for stitching intestinal images from capsule endoscope, and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101716077A (en) * | 2009-12-03 | 2010-06-02 | 西交利物浦大学 | Method and system for processing images based on photographing in vivo by wireless capsule endoscopy or video endoscope |
| US20100165087A1 (en) * | 2008-12-31 | 2010-07-01 | Corso Jason J | System and method for mosaicing endoscope images captured from within a cavity |
| CN102063714A (en) * | 2010-12-23 | 2011-05-18 | 南方医科大学 | Method for generating body cavity full-view image based on capsule endoscope images |
| CN102436665A (en) * | 2011-08-25 | 2012-05-02 | 清华大学 | Two-dimensional plane representation method for images of alimentary tract |
| CN109685721A (en) * | 2018-12-29 | 2019-04-26 | 深圳看到科技有限公司 | Panorama joining method, device, terminal and corresponding storage medium |
| US20190266724A1 (en) * | 2014-06-01 | 2019-08-29 | CapsoVision, Inc. | Reconstruction of Images from an in vivo Multi-Camera Capsule with Two-Stage Confidence Matching |
| CN111524071A (en) * | 2020-04-24 | 2020-08-11 | 安翰科技(武汉)股份有限公司 | Capsule endoscope image splicing method, electronic device and readable storage medium |
| CN113610887A (en) * | 2021-05-26 | 2021-11-05 | 江苏势通生物科技有限公司 | Method for determining capsule endoscope motion shooting path, storage medium and device |
| CN113989125A (en) * | 2021-12-27 | 2022-01-28 | 武汉楚精灵医疗科技有限公司 | Method and device for splicing endoscope images, computer equipment and storage medium |
| CN114066781A (en) * | 2022-01-18 | 2022-02-18 | 浙江鸿禾医疗科技有限责任公司 | Capsule endoscope intestinal tract image identification and positioning method, storage medium and equipment |
| CN114429458A (en) * | 2022-01-21 | 2022-05-03 | 小荷医疗器械(海南)有限公司 | Endoscope image processing method and device, readable medium and electronic equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017218834A1 (en) * | 2016-06-17 | 2017-12-21 | Kerstein Dustin | System and method for capturing and viewing panoramic images having motion parralax depth perception without images stitching |
| CN114972144B (en) * | 2022-05-23 | 2024-02-02 | 江苏势通生物科技有限公司 | Method, device, storage medium and equipment for splicing intestinal images of capsule endoscope |
-
2022
- 2022-05-23 CN CN202210568399.8A patent/CN114972144B/en active Active
-
2023
- 2023-05-22 WO PCT/CN2023/095593 patent/WO2023226941A1/en unknown
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100165087A1 (en) * | 2008-12-31 | 2010-07-01 | Corso Jason J | System and method for mosaicing endoscope images captured from within a cavity |
| CN101716077A (en) * | 2009-12-03 | 2010-06-02 | 西交利物浦大学 | Method and system for processing images based on photographing in vivo by wireless capsule endoscopy or video endoscope |
| CN102063714A (en) * | 2010-12-23 | 2011-05-18 | 南方医科大学 | Method for generating body cavity full-view image based on capsule endoscope images |
| CN102436665A (en) * | 2011-08-25 | 2012-05-02 | 清华大学 | Two-dimensional plane representation method for images of alimentary tract |
| US20190266724A1 (en) * | 2014-06-01 | 2019-08-29 | CapsoVision, Inc. | Reconstruction of Images from an in vivo Multi-Camera Capsule with Two-Stage Confidence Matching |
| CN109685721A (en) * | 2018-12-29 | 2019-04-26 | 深圳看到科技有限公司 | Panorama joining method, device, terminal and corresponding storage medium |
| US20220076376A1 (en) * | 2018-12-29 | 2022-03-10 | Kandao Technology Co., Ltd. | Panoramic image stitching method and apparatus, terminal and corresponding storage medium |
| CN111524071A (en) * | 2020-04-24 | 2020-08-11 | 安翰科技(武汉)股份有限公司 | Capsule endoscope image splicing method, electronic device and readable storage medium |
| CN113610887A (en) * | 2021-05-26 | 2021-11-05 | 江苏势通生物科技有限公司 | Method for determining capsule endoscope motion shooting path, storage medium and device |
| CN113989125A (en) * | 2021-12-27 | 2022-01-28 | 武汉楚精灵医疗科技有限公司 | Method and device for splicing endoscope images, computer equipment and storage medium |
| CN114066781A (en) * | 2022-01-18 | 2022-02-18 | 浙江鸿禾医疗科技有限责任公司 | Capsule endoscope intestinal tract image identification and positioning method, storage medium and equipment |
| CN114429458A (en) * | 2022-01-21 | 2022-05-03 | 小荷医疗器械(海南)有限公司 | Endoscope image processing method and device, readable medium and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| B. ROUSSO ET AL.: "Universal Mosaicing using Pipe Projection", 《SIXTH INTERNATIONAL CONFERENCE ON COMPUTER VISION》, pages 945 - 952 * |
| 刘丽芳: "内窥镜图像拼接", 《中国优秀硕士学位论文全文数据库》, pages 1 - 71 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023226941A1 (en) * | 2022-05-23 | 2023-11-30 | 江苏势通生物科技有限公司 | Method and apparatus for stitching intestinal images from capsule endoscope, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023226941A1 (en) | 2023-11-30 |
| CN114972144B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021213508A1 (en) | Capsule endoscopic image stitching method, electronic device, and readable storage medium | |
| US10198872B2 (en) | 3D reconstruction and registration of endoscopic data | |
| JP7127785B2 (en) | Information processing system, endoscope system, trained model, information storage medium, and information processing method | |
| US10068334B2 (en) | Reconstruction of images from an in vivo multi-camera capsule | |
| Lurie et al. | 3D reconstruction of cystoscopy videos for comprehensive bladder records | |
| CN106056537A (en) | Medical image splicing method and device | |
| CN111798451B (en) | 3D guide wire tracking method and device based on blood vessel 3D/2D matching | |
| EP1685787B1 (en) | Insertion support system | |
| WO2023138544A1 (en) | Capsule endoscope intestinal image-based recognition and positioning method, storage medium, and device | |
| EP3939006A1 (en) | Feature point detection | |
| ES2769636T3 (en) | Automatic 3-D surface coating | |
| EP3148399B1 (en) | Reconstruction of images from an in vivo multi-camera capsule with confidence matching | |
| WO2023226941A1 (en) | Method and apparatus for stitching intestinal images from capsule endoscope, and storage medium | |
| CN113052956B (en) | Method, device and medium for constructing film reading model based on capsule endoscope | |
| WO2023246441A1 (en) | Three-dimensional panoramic recognition and positioning method for digestive tract, storage medium, and computer device | |
| Liu et al. | Global and local panoramic views for gastroscopy: an assisted method of gastroscopic lesion surveillance | |
| CN102063714A (en) | Method for generating body cavity full-view image based on capsule endoscope images | |
| Wang et al. | Robust motion estimation and structure recovery from endoscopic image sequences with an adaptive scale kernel consensus estimator | |
| CN115619790A (en) | Mixed perspective method, system and equipment based on binocular positioning | |
| CN116563118A (en) | Endoscopic image stitching method and device and computer equipment | |
| CN110035687A (en) | The method and device of image mosaic is carried out to the image for using capsule cameras to be captured | |
| KR20210099835A (en) | Method for generating panoramic image and image processing apparatus therefor | |
| US11074672B2 (en) | Method of image processing and display for images captured by a capsule camera | |
| CN117132462A (en) | Endoscopic image stitching method, device, computer equipment and storage medium | |
| CN109754366B (en) | Image splicing method based on binary tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |