WO2023246441A1 - 消化道的三维全景识别定位方法、存储介质和计算机设备 - Google Patents

消化道的三维全景识别定位方法、存储介质和计算机设备 Download PDF

Info

Publication number
WO2023246441A1
WO2023246441A1 PCT/CN2023/097186 CN2023097186W WO2023246441A1 WO 2023246441 A1 WO2023246441 A1 WO 2023246441A1 CN 2023097186 W CN2023097186 W CN 2023097186W WO 2023246441 A1 WO2023246441 A1 WO 2023246441A1
Authority
WO
WIPO (PCT)
Prior art keywords
digestive tract
image
frame
original
projection line
Prior art date
Application number
PCT/CN2023/097186
Other languages
English (en)
French (fr)
Inventor
李鹏
马婷
胡峰
吴建芳
刘揆亮
Original Assignee
江苏势通生物科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 江苏势通生物科技有限公司 filed Critical 江苏势通生物科技有限公司
Publication of WO2023246441A1 publication Critical patent/WO2023246441A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30028—Colon; Small intestine

Definitions

  • the invention belongs to the field of medical equipment imaging technology, and specifically relates to a three-dimensional panoramic recognition and positioning method of the digestive tract, a computer-readable storage medium, and computer equipment.
  • Capsule endoscope is a medical device.
  • the capsule endoscope integrates core functions such as image collection and wireless transmission into a capsule that can be swallowed by the human body. During the examination, the capsule endoscope is swallowed into the body.
  • the endoscope collects images of the digestive tract inside the body and transmits them simultaneously to the outside of the body for medical examination and diagnosis based on the image data obtained.
  • Capsule endoscopy acquires and transmits tens of thousands of images as it works inside the small intestine.
  • the traditional diagnostic method and process is to present these tens of thousands of images to medical workers for observation through picture playback or video playback, and the whole process takes a long time.
  • medical workers cannot visually observe the approximate location of target areas such as suspicious lesions in the digestive tract, nor can they determine the exact location of the target area in the digestive tract, which greatly affects the performance of gastrointestinal endoscopy. usage and diagnostic efficiency.
  • the technical problem to be solved by the present invention is: how to quickly and intuitively observe the target area in the digestive tract and determine the position of the target area.
  • the three-dimensional panoramic recognition and positioning method includes:
  • a new regional image is intercepted from each frame of the original digestive tract image in the original digestive tract image set to form a new regional image set;
  • Each fragment in the collection of three-dimensional tubular inner wall fragments is spliced end-to-end in order to form a three-dimensional panoramic image of the digestive tract;
  • the position information of the target area is determined based on the coordinates of the three-dimensional tubular inner wall segment where the target area is located, the structural length of each three-dimensional tubular inner wall segment, and frame rotation information.
  • the capsule endoscope is provided with an attitude sensor, and the method of obtaining the frame rotation information of each frame of the original digestive tract image in the original digestive tract image set relative to the adjacent previous frame of the original digestive tract image includes:
  • attitude information set collected by the attitude sensor and the fixed sampling time of the attitude sensor, wherein the attitude information set includes the rotation angular velocity in the three-axis direction at multiple consecutive moments;
  • the frame rotation information of each frame of the original digestive tract image relative to the adjacent previous frame of the original digestive tract image is calculated based on the posture information set and the fixed sampling time.
  • the method of obtaining the frame rotation information of each frame of the original digestive tract image in the original digestive tract image set relative to the adjacent previous frame of the original digestive tract image includes:
  • An essential matrix is constructed based on the pixel coordinate values of each feature matching point
  • the frame rotation matrix that meets the preset conditions is determined based on the pixel coordinate values of one pair of feature matching points, the frame rotation matrix, and several estimated values of the translation matrix, as the frame rotation information.
  • the method of intercepting the newly added region image on each frame of the original digestive tract image in the original digestive tract image set based on the structural parameters of the capsule endoscope and the frame rotation information of the corresponding frame includes:
  • the image between the marker projection line and the imaging projection line in the original digestive tract image of the current frame is intercepted as a new area image.
  • the method of determining the imaging projection line of the landmark projection line of the original digestive tract image of the previous frame in the original digestive tract image of the current frame according to the structural parameters of the capsule endoscope and the posture conversion value includes:
  • the candidate projection line corresponding to the Levenstein distance that meets the preset conditions is used as the imaging projection line.
  • the three-dimensional tubular inner wall segment is a chamfered cylindrical cylinder with both ends open, the top surface of the chamfered cylindrical cylinder is a plane, and the bottom end of the chamfered cylindrical cylinder is a sloped surface.
  • the method of constructing and obtaining the three-dimensional tubular inner wall segment based on the structural parameters of the capsule endoscope, the frame rotation information and the newly added area image includes:
  • the diameter, inclination angle and length of the beveled cylindrical barrel are calculated according to the structural parameters and frame rotation information of the capsule endoscope;
  • the pixel value of the inner wall surface of the beveled cylinder is calculated based on the pixel value of the newly added area image.
  • the method of sequentially splicing each segment in the set of three-dimensional tubular inner wall segments head to tail includes:
  • each chamfered cylindrical cylinder is deformed so that the surface of the top of each chamfered cylindrical cylinder is adjusted to match the surface of the bottom end of the next adjacent chamfered cylindrical cylinder, forming a deformed A three-dimensional tubular inner wall segment;
  • each deformed three-dimensional tubular inner wall segment is connected end to end, and each deformed three-dimensional tubular inner wall segment is rotated according to each corresponding attitude conversion value to form a three-dimensional panoramic image of the digestive tract.
  • This application also discloses a computer-readable storage medium that stores a three-dimensional panoramic recognition and positioning program for the digestive tract.
  • the three-dimensional panoramic recognition and positioning program for the digestive tract is executed by a processor, the above-mentioned digestion process is realized.
  • This application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a three-dimensional panoramic recognition and positioning program of the digestive tract stored in the computer-readable storage medium.
  • a computer device which includes a computer-readable storage medium, a processor, and a three-dimensional panoramic recognition and positioning program of the digestive tract stored in the computer-readable storage medium.
  • the panoramic recognition and positioning program is executed by the processor, the above-mentioned three-dimensional panoramic recognition and positioning method of the digestive tract is implemented.
  • the invention discloses a three-dimensional panoramic recognition and positioning method, storage medium and computer equipment of the digestive tract. Compared with traditional methods, it has the following technical effects:
  • Figure 1 is a flow chart of a three-dimensional panoramic recognition and positioning method of the digestive tract according to Embodiment 1 of the present invention
  • Figure 2 is a schematic diagram of the measurement components in the capsule endoscope according to Embodiment 1 of the present invention.
  • Figure 3 is a schematic diagram of an original digestive tract image according to Embodiment 1 of the present invention.
  • Figure 4 is a schematic diagram of the imaging principle of the capsule endoscope according to Embodiment 1 of the present invention.
  • Figure 5 is a schematic diagram of projection lines in the original digestive tract image according to Embodiment 1 of the present invention.
  • Figure 6 is a schematic diagram of posture conversion of the capsule endoscope according to Embodiment 1 of the present invention.
  • Figure 7 is a schematic diagram of a three-dimensional tubular inner wall segment according to Embodiment 1 of the present invention.
  • Figure 8 is a schematic diagram of the pixel parameter calculation process of the three-dimensional tubular inner wall segment according to Embodiment 1 of the present invention.
  • Figure 9 is a schematic diagram of a three-dimensional tubular inner wall segment undergoing deformation processing according to Embodiment 1 of the present invention.
  • Figure 10 is a schematic diagram of the splicing process of each three-dimensional tubular inner wall segment according to Embodiment 1 of the present invention.
  • Figure 11 is a schematic process diagram of the three-dimensional panoramic recognition and positioning method of the digestive tract according to Embodiment 1 of the present invention.
  • Figure 12 is a schematic diagram of computer equipment according to Embodiment 3 of the present invention.
  • identification and diagnosis are based on intestinal images captured by capsule endoscopes, mainly through video or image reading.
  • the number of images is large, the entire reading process takes a long time, and the approximate location of the target area in the digestive tract cannot be visually observed and the precise location of the target area cannot be determined.
  • this application provides a three-dimensional panoramic recognition and positioning method for the digestive tract.
  • the main step of this method is to intercept a new area image on each frame of the original digestive tract image, and combine the structural parameters of the capsule endoscope and the original two adjacent frames.
  • the frame rotation information between the digestive tract images is reconstructed to obtain three-dimensional tubular inner wall segments.
  • Each three-dimensional tubular inner wall segment is spliced to obtain a three-dimensional panoramic image of the digestive tract. After the target area is identified in the three-dimensional panoramic image of the digestive tract, the position information of the target area is further determined. .
  • the whole process of image reading and diagnosis using this method is relatively intuitive and convenient, and it can quickly and accurately determine the location of the target area.
  • the three-dimensional panoramic recognition and positioning method of the digestive tract in this embodiment includes the following steps:
  • Step S10 Obtain the frame rotation information of each frame of the original digestive tract image in the set of original digestive tract images captured by the capsule endoscope relative to the adjacent previous frame of the original digestive tract image;
  • Step S20 Based on the structural parameters and frame rotation information of the capsule endoscope, a new region image is intercepted from each frame of the original digestive tract image in the original digestive tract image set to form a new region image set;
  • Step S30 Construct a three-dimensional tubular inner wall segment set, wherein each segment in the three-dimensional tubular inner wall segment set is constructed based on the structural parameters of the capsule endoscope, the frame rotation information and the newly added area image;
  • Step S40 Splice the fragments in the collection of three-dimensional tubular inner wall fragments head-to-tail sequentially to form a three-dimensional panoramic image of the digestive tract;
  • Step S50 After identifying the target area in the three-dimensional panoramic image of the digestive tract, determine the three-dimensional tubular inner wall segment where the target area is located;
  • Step S60 Determine the position information of the target area based on the coordinates of the three-dimensional tubular inner wall segment where the target area is located, the structural length of each three-dimensional tubular inner wall segment, and frame rotation information.
  • the measurement component 10 on the capsule endoscope consists of a capsule transparent cover 101 and an image acquisition module 103 located within the capsule arc dividing line 102a.
  • Some structural parameters of capsule endoscopes include glue
  • the inner diameter of the capsule is R1, the outer diameter is R2, and the central axis is a.
  • the viewing angle extension line of the image acquisition module 103 intersects with the capsule shell at 102b.
  • the imaging optical center of the image acquisition module 103 is O, and the internal parameters of the imaging model are K.
  • the capsule endoscope collects images in the digestive tract and stores or sends the collected data to the user.
  • the in vitro equipment carried by the patient will process the data collected during the examination after completing the examination.
  • the data to be processed is a series of original digestive tract image sets ⁇ I n ⁇ captured by the image acquisition module 103, 1 ⁇ n ⁇ N and posture data ⁇ m collected at a fixed sampling time TS .
  • each frame of the original digestive tract image in the original digestive tract image set is shown as 20 in Figure 3, where the image sensor frame, that is, the maximum imaging area 201, determines the maximum image that the image sensor can collect, and the maximum imaging area 201 has The imaging projection boundary 202 of the capsule shell 101 on the image sensor, and the imaging projection boundary 203 of the capsule arc-shaped dividing line 102a on the image sensor.
  • the original digestive tract image 20 has two orthogonal plane direction axes u and v, respectively along the two rectangular sides of the single-frame original digestive tract image 20, and in the single-frame original The center of the digestive tract image 20 is the origin.
  • the measurement component 10 also integrates an attitude sensor 104, and the attitude sensor 104 is preferably a gyroscope.
  • the calibrated attitude information set ⁇ m ⁇ collected by the attitude sensor 104, 1 ⁇ m ⁇ M, the total inspection running time is MT S .
  • Attitude information set ⁇ m ⁇ , each unit data in 1 ⁇ m ⁇ M contains the rotation angular velocity in the three-axis direction and has the following form:
  • ⁇ u,m is the rotation angular velocity of the u-axis obtained by the attitude sensor 104
  • ⁇ v,m is the rotation angular velocity of the v-axis obtained by the attitude sensor 104
  • ⁇ a,m is the rotation angular velocity of the capsule central axis a-axis obtained by the attitude sensor 104. Rotation angular velocity.
  • a single frame image in the original digestive tract image set ⁇ I n ⁇ has a timestamp ⁇ s n ⁇ , 1 ⁇ n ⁇ N, 1 ⁇ s n ⁇ M, that is, the original digestive tract image frame I n has a timestamp s n , and the image
  • the collection time of In is identified as s n T S .
  • step S10 there are two methods for obtaining frame rotation information. One is to obtain it based on the data collected by the attitude sensor 104 , and the other is to obtain it through feature matching between images.
  • the m-th acquisition of the attitude sensor 104 is calculated.
  • the matching method can be SIFT, SURF or ORB algorithm, and RANSAC is used to filter the obtained matching feature points, and finally P pairs of matching feature points ⁇ p i, n-1 ⁇ , 1 ⁇ i ⁇ P and ⁇ p i, n ⁇ , 1 ⁇ i ⁇ P, respectively belong to the previous frame of the original digestive tract image I n-1 and the current frame of the original digestive tract image frame I n .
  • the image pixel coordinates of each feature matching point are:
  • U is the left singular matrix of E
  • U T is the right singular matrix of E
  • ⁇ represents the characteristic matrix of the essential matrix E
  • R a ( ⁇ ) represents the rotation matrix of the rotation angle ⁇ along the a-axis.
  • step S20 the method of intercepting a new region image on each frame of the original digestive tract image in the original digestive tract image set based on the structural parameters of the capsule endoscope and the frame rotation information of the corresponding frame includes the following steps:
  • Step S201 Determine the marker projection line in each frame of the original digestive tract image according to the structural parameters of the capsule endoscope
  • Step S202 Calculate the posture conversion value between the original digestive tract image of the current frame and the adjacent original digestive tract image of the previous frame based on the frame rotation information of the original digestive tract image of the current frame to be intercepted;
  • Step S203 Determine the imaging projection line of the marker projection line of the original digestive tract image of the previous frame in the original digestive tract image of the current frame according to the structural parameters of the capsule endoscope and the posture conversion value;
  • Step S204 Intercept the image between the marker projection line and the imaging projection line in the original digestive tract image of the current frame as a new area image.
  • Figure 4 shows the imaging process of images collected by a capsule endoscope, including the capsule arc-shaped dividing line 102a, the intersection line 102b of the image acquisition module's viewing angle and the capsule shell, and the imaging plane 103a of the image acquisition module 103.
  • the optical center of the image acquisition module is point O, and the glue
  • the intersection point of the capsule axis a-axis with 102a is E
  • the intersection point with 102b is D
  • the intersection point with the imaging plane 103a is C.
  • the capsule endoscope has no deflection and the resulting projection line is circular. In actual situations, due to the deflection of the capsule endoscope, the resulting projection line is elliptical.
  • the capsule endoscope adheres to the inner wall of the digestive tract such as the intestine during its travel, that is, the two-point FX adheres to the inner wall of the digestive tract.
  • the imaging of the two-point FX is equivalent to the imaging of the inner wall of the digestive tract.
  • the imaging principle is also the same. In order to simplify the description here, the imaging process is explained by taking the imaging of dots on the transparent cover as an example.
  • the landmark projection line in the original digestive tract image is preferably the projection of the capsule arc-shaped dividing line 102a on the imaging plane.
  • the marker projection of the original digestive tract image I n-1 in the previous frame The hatch line is 301, and the marker projection line of the original digestive tract image In of the current frame is 303.
  • 302 represents the imaging of the marker projection line 301 of the original digestive tract image In -1 of the previous frame in the original digestive tract image In of the current frame. Projection line.
  • the shooting direction of the capsule endoscope in Embodiment 1 is the same as the moving direction, that is, when the capsule endoscope is advancing, the inner wall of the intestine in front of it is photographed, so the imaging projection line 302 is projected on the sign. Outside line 301 or 303.
  • the technical method described in this manual is also applicable to the situation where the shooting direction of the capsule endoscope is opposite to the moving direction.
  • the imaging projection line 302 takes a quasi-elliptical shape, so it is necessary to determine the imaging projection line 302 by combining the frame rotation information between images.
  • the frame rotation matrix from the previous frame of the original digestive tract image I n-1 to the current frame of the original digestive tract image I n is R T, n , which corresponds to the time of the previous frame of the original digestive tract image I n-1
  • the capsule axis a is rotated to the vector R.
  • the angle between the vector R and the capsule axis a at time In -1 is ⁇ n .
  • the angle between the projection of the vector R on the uv plane and the u-axis is
  • step S202 the attitude conversion value includes the above-mentioned ⁇ n and
  • the original digestive tract image In of the current frame has a center point C, and an angle with the u-axis along the center point C.
  • the straight line intersects the imaging projection line at points P and Q.
  • point P to have a distance parameter h t , and define the projection curve equation as a polar coordinate form with point C as the origin.
  • point P to have a distance parameter h t , and define the projection curve equation as a polar coordinate form with point C as the origin. Therefore, the polar coordinate equation of the imaging projection line 302 under the condition of distance parameter h t is:
  • step S203 the process of determining the imaging projection line is as follows:
  • Step S2031 Determine several candidate projection lines of the marker projection line of the original digestive tract image of the previous frame in the original digestive tract image of the current frame according to the structural parameters of the capsule endoscope and the posture conversion value.
  • the size of the candidate projection line search space be SI.
  • the larger the SI the longer the search time.
  • Step S2032 Calculate the marker projection line of the original digestive tract image of the previous frame and the discrete quantized pixel gray value sequence of each candidate projection line;
  • the logo projection line is a discrete quantized pixel gray value sequence E n-1 of 301.
  • the calculation formula is as follows:
  • E n-1 ⁇ Q n-1, k mod Q ⁇ , 1 ⁇ k ⁇ K, where K is the quantization accuracy.
  • K the quantization accuracy.
  • K 128 .
  • Q is a quantification scale, 1 ⁇ Q ⁇ 255.
  • a larger Q value corresponds to faster calculation efficiency, but the accuracy will decrease.
  • Q 26.
  • Step S2033 Sequentially calculate the Levinstein distance between the discrete quantized pixel gray value sequence of the landmark projection line of the previous frame of the original digestive tract image and the discrete quantized pixel gray value sequence of each candidate projection line.
  • Lev(a, b) represents the calculation of Levenstein distance between sequences a and b.
  • Step S2034 Use the candidate projection line corresponding to the Levenstein distance that meets the preset conditions as the imaging projection line.
  • the imaging projection line satisfies the following conditions:
  • Lev(E n-1 ,E n,s ) ⁇ Lev th or Lev(E n-1 ,E n,s ) min(Lev(E n-1 ,E n,i )), 1 ⁇ i ⁇ SI
  • step S204 the image between the marker projection line 303 and the imaging projection line 302 in the original digestive tract image In of the current frame is intercepted as a new area image I C,n , forming a new area image set ⁇ I C,n ⁇ .
  • step S30 the reconstruction of the three-dimensional tubular inner wall segment mainly includes the construction of the structural parameters and pixel parameters of the three-dimensional tubular inner wall segment.
  • the three-dimensional tubular inner wall segment is a chamfered cylindrical cylinder with both ends open.
  • the top surface of the chamfered cylindrical cylinder is a flat surface
  • the bottom surface of the chamfered cylindrical cylinder is a sloped surface.
  • the three-dimensional tubular inner wall fragment photographed should be a right cylinder.
  • the three-dimensional tubular inner wall fragment photographed should be as shown in Figure 7.
  • the structural parameters of the three-dimensional tubular inner wall segment include the diameter of the beveled cylindrical barrel, the inclination angle ⁇ n and the length of the beveled cylindrical barrel.
  • the surface where the top of the beveled cylindrical cylinder is located is a plane, and the plane is a circular top surface centered on O 2.
  • the surface where the bottom end of the beveled cylindrical cylinder is located is an inclined plane, and the inclined surface is an elliptical bottom surface centered on O 1 .
  • the major axis vertices of the ellipse base are P 1 and Q 1 respectively, and the corresponding cylindrical side edges are P 1 P 2 and Q 1 Q 2 respectively. in:
  • the diameter of the chamfered cylinder is 2R 0 and the length of the chamfered cylinder is D n .
  • Pixel value p n (u, v) of the newly added area image and the pixel value of the X point on the inner wall of the beveled cylinder Pixel values have the following relationship:
  • the three-dimensional tubular inner wall segment J n is obtained, forming a three-dimensional tubular inner wall segment set ⁇ J n ⁇ .
  • step S40 the side wall of each chamfered cylindrical cylinder is first deformed so that the top surface of each chamfered cylindrical cylinder is adjusted to be in line with the surface of the adjacent chamfered cylindrical cylinder.
  • the faces where the bottom ends are located match to form a deformed three-dimensional tubular inner wall segment.
  • Figure 9 shows a cross-sectional view of the three-dimensional tubular inner wall segment J n described in Figure 8 along the P 1 P 2 Q 2 Q 1 plane, where a n is the axis a n+1 of the moment capsule of the original digestive tract image I n of the current frame. It is the axis axis of the capsule at time I n+1 of the original digestive tract image of the next frame.
  • the two image frames have a rotation matrix R T, n+1 , corresponding to the axis axis deflection angle ⁇ n+1 generated by the posture conversion.
  • the deformation process is to transform the top plane 401 of the three-dimensional tubular inner wall segment Jn into the top plane 402 of the deformed three-dimensional tubular inner wall segment Un .
  • the center point O2 of the two planes remains unchanged.
  • the pixels on the side walls of the three-dimensional tubular inner wall segment are stretched and compressed according to the expansion and contraction of the edges.
  • the stretch deformation is performed by pixel interpolation filling.
  • each deformed three-dimensional tubular inner wall segment is connected end to end, and each pair of deformed three-dimensional tubular inner wall segments is rotated according to each corresponding attitude conversion value to form a three-dimensional panoramic image of the digestive tract.
  • three-dimensional tubular inner wall segments U n-1 and U n are selected, which have lower bottom surface centers On -1, 1 and On , 1 respectively, and upper top surface centers On -1, 2 and On ,2 .
  • Deflect the axis a n of U n relative to the axis a n-1 of U n-1 by an angle ⁇ n and deflect the u-axis of U n relative to the u-axis of U n-1 angle.
  • all ⁇ U n ⁇ operations are completed to generate a three-dimensional panoramic image of the digestive tract.
  • step S50 after the user performs lesion identification and judgment through translation, rotation and zooming, the target area is determined.
  • the location where the target area is located can be determined.
  • a spatial coordinate system is established, which is a Cartesian coordinate system with the center of the original digestive tract image I 1 of the first frame as the origin. Calculate the coordinates of the lower bottom center O n,1 of the three-dimensional tubular inner wall segment U n in the spatial coordinate system as the global coordinates:
  • the local coordinates of the target area X can be calculated based on the three-dimensional tubular inner wall segment shown in Figure 8.
  • the local polar coordinates of the target area X are Then the local Cartesian coordinates of the target area X are
  • Figure 11 shows the overall process from processing the original digestive tract image to finally locating the target area.
  • the second embodiment also discloses a computer-readable storage medium.
  • the computer-readable storage medium stores a three-dimensional panoramic recognition and positioning program of the digestive tract.
  • the three-dimensional panoramic recognition and positioning program of the digestive tract is executed by the processor, the above digestion is realized.
  • the third embodiment also discloses a computer device.
  • the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11.
  • the processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level.
  • one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each A logic unit can also be a hardware or logic device.
  • the computer-readable storage medium 11 stores a three-dimensional panoramic recognition and positioning program for the digestive tract. When the three-dimensional panoramic recognition and positioning program for the digestive tract is executed by the processor, the above-mentioned three-dimensional panoramic recognition and positioning method for the digestive tract is implemented.
  • Computer-readable storage media includes permanent and non-transitory, removable and non-removable media and may be implemented by any method or technology to store information. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory Memory (ROM), electrically erasable programmable only read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic disk storage, quantum memory, Graphene-based storage media or other magnetic storage devices, or any other non-transmission media, can be used to store information that can be accessed by computing devices.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • ROM read-only memory Memory
  • EEPROM electrically eras

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Endoscopes (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

一种消化道的三维全景识别定位方法、存储介质和计算机设备。该方法包括：获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息（S10）；基于胶囊内窥镜的结构参数和帧旋转信息在每一帧原始消化道图像上截取新增区域图像，构成新增区域图像集（S20）；构建三维管状内壁片段集，其中三维管状内壁片段集中每个片段根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到（S30）；将三维管状内壁片段集中的各个片段依序进行首尾拼接，形成消化道三维全景图像（S40）；确定目标区域所在的三维管状内壁片段并确定目标区域的位置信息（S50）。该方法的阅片、诊断的整个过程比较直观、便捷，且能快速、精准地确定目标区域的位置。

Description

消化道的三维全景识别定位方法、存储介质和计算机设备 技术领域
本发明属于医疗设备成像技术领域，具体地讲，涉及一种消化道的三维全景识别定位方法、计算机可读存储介质、计算机设备。
背景技术
胶囊内窥镜是一种医疗器械设备，胶囊内窥镜将图像采集、无线传输等核心功能集成于一个可被人体吞咽的胶囊内，在进行检查过程中，将胶囊内窥镜吞入体内，内窥镜在体内采集消化道图像并同步传送到体外，以根据获得的图像数据进行医疗检查和诊断。
胶囊内窥镜在小肠内工作过程中会采集并传输数万张图像。传统的诊断方法和流程是将这数万张图像通过图片播放或者视频播放的方式呈现给医务工作者进行观察，整个过程耗时长。同时，对二维图像进行阅片，医务工作者也无法直观地观察可疑病灶等目标区域在消化道中的大致位置，更无法确定目标区域在消化道中的准确位置，极大影响了消化道内窥镜的使用和诊断效率。
发明内容
本发明所要解决的技术问题是：如何快速、直观地观察消化道中的目标区域并确定该目标区域的位置。
本申请公开了一种消化道的三维全景识别定位方法，所述三维全景识别定位方法包括：
获取胶囊内窥镜拍摄到的原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息；
基于胶囊内窥镜的结构参数和帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像，构成新增区域图像集；
构建三维管状内壁片段集，其中所述三维管状内壁片段集中每个片段根据 胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到；
将所述三维管状内壁片段集中的各个片段依序进行首尾拼接，形成消化道三维全景图像；
在所述消化道三维全景图像中识别出目标区域后，确定目标区域所在的三维管状内壁片段；
根据所述目标区域位于所在三维管状内壁片段的坐标、各个所述三维管状内壁片段的结构长度、帧旋转信息，确定所述目标区域的位置信息。
可选地，所述胶囊内窥镜上设置有姿态传感器，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
获取姿态传感器采集到的姿态信息集合和姿态传感器的固定采样时间，其中所述姿态信息集合包括连续多个时刻的三轴方向上的旋转角速度；
基于所述姿态信息集合和所述固定采样时间计算得到每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息。
可选地，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
获取每一帧原始消化道图像与前一帧原始消化道图像进行特征匹配，获得若干对特征匹配点；
基于各个所述特征匹配点的像素坐标值构建得到本质矩阵；
基于所述本质矩阵分别得到帧旋转矩阵和平移矩阵的若干个估计值；
根据若干对特征匹配点的其中一对特征匹配点的像素坐标值、帧旋转矩阵和平移矩阵的若干个估计值确定符合预设条件的帧旋转矩阵，作为帧旋转信息。
可选地，基于胶囊内窥镜的结构参数和对应帧的帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像的方法包括：
根据胶囊内窥镜的结构参数确定每一帧原始消化道图像中的标志投影线；
根据待截取的当前帧原始消化道图像的帧旋转信息计算得到当前帧原始消化道图像与相邻的前一帧原始消化道图像之间的姿态转换值；
根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道 图像的标志投影线在所述当前帧原始消化道图像中的成像投影线；
截取所述当前帧原始消化道图像中标志投影线和成像投影线之间的图像，作为新增区域图像。
可选地，根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线的方法包括：
根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的若干候选投影线；
计算前一帧原始消化道图像的标志投影线以及各所述候选投影线的离散量化像素灰度值序列；
依次计算前一帧原始消化道图像的标志投影线的离散量化像素灰度值序列与各所述候选投影线的离散量化像素灰度值序列之间的莱文斯坦距离；
将符合预设条件的莱文斯坦距离所对应的候选投影线作为成像投影线。
可选地，所述三维管状内壁片段为两端开放的斜切圆柱筒体，所述斜切圆柱筒体的顶端所在面为平面，所述斜切圆柱筒体的底端所在面为斜面。
可选地，根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到所述三维管状内壁片段的方法包括：
根据所述胶囊内窥镜的结构参数、帧旋转信息计算得到所述斜切圆柱筒体的直径、倾斜角度以及斜切圆柱筒体的长度；
根据所述新增区域图像的像素值计算得到所述斜切圆柱筒体的内壁面的像素值。
可选地，将所述三维管状内壁片段集中的各个片段依序进行首尾拼接的方法包括：
将每个斜切圆柱筒体的侧壁进行变形处理，使得每个斜切圆柱筒体的顶端所在面调整为与相邻的下一个斜切圆柱筒体的底端所在面匹配，形成变形后的三维管状内壁片段；
将各个变形后的三维管状内壁片段的顶端和底端首尾连接，并且按照各个对应的姿态转换值对各个变形后的三维管状内壁片段进行旋转，形成消化道三维全景图像。
本申请还公开了一种计算机可读存储介质，所述计算机可读存储介质存储有消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现上述的消化道的三维全景识别定位方法。
本申请还公开了一种计算机设备，所述计算机设备包括计算机可读存储介质、处理器和存储在所述计算机可读存储介质中的消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现上述的消化道的三维全景识别定位方法。
本发明公开的一种消化道的三维全景识别定位方法、存储介质和计算机设备，相对于传统方法，具有如下技术效果：
通过将原始肠道图片拼接形成肠道全景图片，使得阅片、诊断的整个过程比较直观、便捷，且能快速、精准地确定目标区域的位置。
附图说明
图1为本发明的实施例一的消化道的三维全景识别定位方法的流程图；
图2为本发明的实施例一的胶囊内窥镜中的测量部件示意图；
图3为本发明的实施例一的原始消化道图像的示意图；
图4为本发明的实施例一的胶囊内窥镜的成像原理示意图；
图5为本发明的实施例一的原始消化道图像中的投影线示意图；
图6为本发明的实施例一的胶囊内窥镜的姿态转换示意图；
图7为本发明的实施例一的三维管状内壁片段的示意图；
图8为本发明的实施例一的三维管状内壁片段的像素参数计算过程示意图；
图9为本发明的实施例一的三维管状内壁片段进行变形处理的示意图；
图10为本发明的实施例一的各个三维管状内壁片段的拼接过程示意图；
图11为本发明的实施例一的消化道的三维全景识别定位方法的过程示意图；
图12为本发明的实施例三的计算机设备示意图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
在详细描述本申请的各个实施例之前，首先简单描述本申请的技术构思：现有技术中根据胶囊内窥镜拍摄的肠道图像进行识别和诊断，主要通过视频或图像阅片的方式，由于图像数量较大，整个阅片过程耗时较长，且直观地观察目标区域在消化道中的大致位置以及无法确定目标区域的精确位置。为此，本申请提供的消化道的三维全景识别定位方法，该方法主要步骤是在每帧原始消化道图像上截取新增区域图像，并结合胶囊内窥镜的结构参数、相邻两帧原始消化道图像之间帧旋转信息重建得到三维管状内壁片段，将各个三维管状内壁片段进行拼接得到消化道三维全景图像，在消化道三维全景图像中识别出目标区域后，进一步确定目标区域的位置信息。该方法的阅片、诊断的整个过程比较直观、便捷，且能快速、精准地确定目标区域的位置。
具体来说，如图1所示，本实施例一的消化道的三维全景识别定位方法包括如下步骤：
步骤S10、获取胶囊内窥镜拍摄到的原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息；
步骤S20、基于胶囊内窥镜的结构参数和帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像，构成新增区域图像集；
步骤S30、构建三维管状内壁片段集，其中所述三维管状内壁片段集中每个片段根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到；
步骤S40、将所述三维管状内壁片段集中的各个片段依序进行首尾拼接，形成消化道三维全景图像；
步骤S50、在所述消化道三维全景图像中识别出目标区域后，确定目标区域所在的三维管状内壁片段；
步骤S60、根据所述目标区域位于所在三维管状内壁片段的坐标、各个所述三维管状内壁片段的结构长度、帧旋转信息，确定所述目标区域的位置信息。
在对各个步骤详细说明之前，先对胶囊内窥镜的基本结构和工作过程进行说明。如图2所示，胶囊内窥镜上的测量部件10由胶囊透明盖101、位于胶囊弧形分割线102a以内的图像采集模块103。胶囊内窥镜的部分结构参数包括胶 囊内径R1，外径R2，中心轴线为a。图像采集模块103的视角延伸线与胶囊外壳相交于102b。图像采集模块103的成像光心为O，成像模型的内参数为K。使用者在相应的准备工作完成后，将带有测量部件10的胶囊内窥镜吞服进入消化道，胶囊内窥镜在消化道中进行影像采集，并将采集得到的数据进行存储或发送给使用者携带的体外设备，完成检查后，将检查过程中采集到的数据进行处理。所述待处理的数据为一系列由图像采集模块103拍摄的原始消化道图像集{In}，1≤n≤N以及以固定采样时间TS采集的姿态数据ωm。
其中，原始消化道图像集中的每帧原始消化道图像如图3中的20所示，其中图像传感器边框，即最大成像区域201确定了图像传感器所能采集的最大图像，最大成像区域201内具有胶囊外壳101在图像传感器上的成像投影边界202，以及胶囊弧形分割线102a在图像传感器上的成像投影边界203。根据图像传感器103在胶囊中的安装方向，原始消化道图像20具有两个正交的平面方向轴u和v，分别沿着单帧原始消化道图像20的两条矩形边，并以单帧原始消化道图像20的中心为原点。
进一步地，测量部件10中还集成有姿态传感器104，姿态传感器104优选为陀螺仪。由姿态传感器104采集的校准后的姿态信息集合{ωm}，1≤m≤M，总的检查运行时间为MTS。姿态信息集合{ωm}，1≤m≤M中每一单元数据包含三轴方向上的旋转角速度具有如下的形式：
其中，ωu，m为姿态传感器104获取的u轴的旋转角速度，ωv，m为姿态传感器104获取的v轴的旋转角速度，ωa，m为姿态传感器104获取的胶囊中心轴线a轴的旋转角速度。
原始消化道图像集{In}中单帧图像具有时间戳{sn}，1≤n≤N，1≤sn≤M，即原始消化道图像帧In具有时间戳sn，将图像In的采集时间标识为snTS。
在步骤S10中，获取帧旋转信息的方法有两种，一是基于姿态传感器104采集到的数据来获取，二是通过图像之间的特征匹配来获得。
对于第一种方式，首先基于采集的{ωm}数据，计算姿态传感器104第m次采 样时，相对于第m-1次采样的胶囊姿态旋转矩阵：
其中，σ＝|ωmTS|，
接着，计算原始消化道图像集{In}中，第n张图像相对于第n-1张图像的胶囊姿态旋转转矩阵{RT，n}，1≤n≤N，这里的旋转转矩阵即为帧旋转信息。其中，
对于第二种方式，首先获取每一帧原始消化道图像In与前一帧原始消化道图像In-1进行特征匹配，获得若干对特征匹配点。具体可采用匹配方法可采用SIFT、SURF或ORB算法，并采用RANSAC对获得的匹配特征点进行筛选，最终获得P对匹配特征点{pi，n-1}，1≤i≤P和{pi，n}，1≤i≤P，分别属于前一帧原始消化道图像In-1和当前帧原始消化道图像帧In，各特征匹配点的图像像素坐标为：
进一步地，基于各个所述特征匹配点的像素坐标值构建得到本质矩阵E。具体来说，令

e＝[e1 e2 e3 e4 e5 e6 e7 e8 e9]T
使用最小二乘法求解e，即
构建本质矩阵E，即
接着，基于本质矩阵分别得到帧旋转矩阵R和平移矩阵t的若干个估计值：t＝±URa(±90°)∑UT
其中，U为E的左奇异矩阵，UT为E的右奇异矩阵，∑表示本质矩阵E的特征矩阵，Ra(γ)表示沿a轴旋转γ角度的旋转矩阵。
最后，根据若干对特征匹配点的其中一对特征匹配点的像素坐标值、帧旋转矩阵和平移矩阵的若干个估计值确定符合预设条件的帧旋转矩阵，作为帧旋转信息。具体来说，从P对匹配特征点筛选任意一对，并与帧旋转矩阵R和平移矩阵t构造如下形式的矩阵：
λi＝(Rpi，n-1+t)/pi，n
将帧旋转矩阵R和平移矩阵t的不同估计值代入到上式中，当若获取的矩阵λi的所有元素为正时，则所选取的R、t保留作为第n张图像帧的帧旋转矩阵和平移矩阵RT，n、tn，RT，1＝1，即此时的帧旋转矩阵RT，n作为帧旋转信息。
在步骤S20中，基于胶囊内窥镜的结构参数和对应帧的帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像的方法包括如下步骤：
步骤S201、根据胶囊内窥镜的结构参数确定每一帧原始消化道图像中的标志投影线；
步骤S202、根据待截取的当前帧原始消化道图像的帧旋转信息计算得到当前帧原始消化道图像与相邻的前一帧原始消化道图像之间的姿态转换值；
步骤S203、根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线；
步骤S204、截取所述当前帧原始消化道图像中标志投影线和成像投影线之间的图像，作为新增区域图像。
首先对原始消化道图像中的投影线进行说明。如图4示出了胶囊内窥镜采集图像的成像过程，其中胶囊弧形分割线102a，图像采集模块视角与胶囊外壳交线102b，图像采集模块103的成像平面103a。图像采集模块光心为O点，胶 囊轴线a轴与102a交点为E，与102b交点为D，与成像平面103a交点为C。
令CA＝r1，CB＝r2，OC＝h0，OD＝H0，DE＝HD。
则有如下关系：
可以解得：
则对于胶囊外壳上位于102a与102b之间的任意点X，具有在103a上的成像点Y，令Cy＝rx，FX＝hx，则进而求得
rx表示投影线上的投影点到图像中心C的距离(或者说是投影圆的半径)，反映了图4中的两点FX与投影圆半径之间的关系r＝g(hx)。换句话说，知道了hx就可以算出投影点所在投影圆的半径rx。
需要说明的是，在理想情况下，胶囊内窥镜无偏转，得到的投影线是圆形，实际情况中由于胶囊内窥镜发生偏转，形成的投影线为类椭圆形。同时还需要说明的是，胶囊内窥镜在行进过程中与肠道等消化道的内壁贴合，即两点FX与消化道内壁贴合，两点FX的成像相当于是消化道内壁的成像，当然若胶囊内窥镜为贴合内壁，即两点FX与内壁间隔，其成像原理也是相同的。这里为了简化描述，以透明盖上的点的成像来说说明成像过程。
示例性地，本实施例一原始消化道图像中的标志投影线优选为胶囊弧形分界线102a在成像平面的投影。如图5所示，前一帧原始消化道图像In-1的标志投 影线为301，当前帧原始消化道图像In的标志投影线为303。302表示前一帧原始消化道图像In-1的标志投影线301在当前帧原始消化道图像In中的成像投影线。需要说明的是，本实施例一的胶囊内窥镜的拍摄方向与移动方向相同，即胶囊内窥镜在前进过程中，对其前方的肠道内壁进行拍摄，因此成像投影线302在标志投影线301或303之外。本说明书所述的技术方法同样适用于胶囊内窥镜的拍摄方向与移动方向相反的情形。同时由于胶囊内窥镜在拍摄过程中发生偏转，成像投影线302呈类椭圆形，因此需要结合图像之间的帧旋转信息来确定成像投影线302。
如图6所示，前一帧原始消化道图像In-1到当前帧原始消化道图像In的帧旋转矩阵为RT，n，对应将前一帧原始消化道图像In-1时刻的胶囊轴线轴a旋转至向量R，向量R与In-1时刻胶囊轴线轴a的夹角为θn，向量R在uv平面投影与u轴夹角为
令帧旋转矩阵RT，n的各个元素分别是：
则可得
在步骤S202中姿态转换值包括上述的θn和
在图4中，当前帧原始消化道图像In具有中心点C，沿中心点C具有与u轴夹角的直线，与成像投影线相交于P点和Q点。定义P点具有距离参数ht，定义所述投影曲线方程为以C点为原点的极坐标形式定义P点具有距离参数ht，定义投影曲线方程为以C点为原点的极坐标形式因此，成像投影线302在距离参数ht条件下的极坐标方程为：
其中，在步骤S203中，确定成像投影线的过程如下：
步骤S2031、根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的若干候选投影线。
具体来讲，令候选投影线搜寻空间大小为SI，SI越大，搜寻时间越长，SI越小，搜寻精确度越低。示例性地，设置SI＝20，则令则若干候选投影线构成的集合为：
其中，
步骤S2032、计算前一帧原始消化道图像的标志投影线以及各所述候选投影线的离散量化像素灰度值序列；
标志投影线为301的离散量化像素灰度值序列En-1，计算公式如下：
En-1＝{Qn-1，kmod Q}，1≤k≤K，其中K为量化精度，越大的K对应越精确的搜索，但是运算速度会下降，优选地，K＝128。
Qn-1，k为前一帧原始消化道图像In-1中的像素pn-1，k＝[un-1，k vn-1，k]T的灰度值，
且满足Q为量化刻度，1≤Q≤255，越大的Q值对应越快的计算效率，但是精度会下降，优选的，Q＝26。
计算当前帧原始消化道图像In中，位于候选投影线上的像素点的离散量化像素灰度序列En,i：
En，i＝{Qn，i，kmod Q}，其中Qn，i，k为当前帧原始消化道图像In中的像素pn，i，k的灰度值，且满足
步骤S2033、依次计算前一帧原始消化道图像的标志投影线的离散量化像素灰度值序列与各所述候选投影线的离散量化像素灰度值序列之间的莱文斯坦距离。
通过莱文斯坦距离来评估候选投影线与标志投影线之间的相似性，
其中，Lev(a，b)表示计算序列a和b的莱文斯坦距离。
步骤S2034、将符合预设条件的莱文斯坦距离所对应的候选投影线作为成像投影线。
具体，设置相似性阈值Levth，选择候选投影线集合其中之一的投影线作为成像投影线，且成像投影线满足如下条件：
Lev(En-1，En，s)≤Levthor Lev(En-1，En，s)＝min(Lev(En-1，En，i))，1≤i≤SI
在步骤S204截取当前帧原始消化道图像In中标志投影线303和成像投影线302之间的图像作为新增区域图像IC，n，构成新增区域图像集{IC，n}。
在步骤S30中，三维管状内壁片段的重建主要包括三维管状内壁片段的结构参数和像素参数两部分的构建。
具体来说，如图7所示，三维管状内壁片段为两端开放的斜切圆柱筒体，斜切圆柱筒体的顶端所在面为平面，斜切圆柱筒体的底端所在面为斜面。在理想情况下，当胶囊内窥镜无偏转时，拍摄的三维管状内壁片段应为正圆筒体，当胶囊内窥镜偏向一侧时，拍摄得到三维管状内壁片段为图7所示的一侧较长另一侧较短的斜切圆柱筒体。
其中，三维管状内壁片段的结构参数包括斜切圆柱筒体的直径、倾斜角度θn以及斜切圆柱筒体的长度。斜切圆柱筒体的顶端所在面为平面，平面为以O2圆心的圆形顶面，斜切圆柱筒体的底端所在面为斜面，斜面为以具有以O1为中心的椭圆底面。椭圆底面的长轴顶点分别为P1和Q1，与其对应的圆柱侧棱分别为P1P2以及Q1Q2。其中：
Q2P2＝Q2Q2＝R0，P1P2＝ln，t，Q1Q2＝ln，t+2R0tanθn，
在上式中，斜切圆柱筒体的直径为2R0，斜切圆柱筒体的长度为Dn。
对于斜切圆柱筒体的柱面上的任意点X，其在以O2为圆心的圆形顶面投影为X’，O3P2与O2X′的夹角为点X的坐标为且满足
根据所述新增区域图像的像素值pn(u，v)与斜切圆柱筒体内壁面的X点的像素值像素值具有如下关系：
且其中，
因此，计算上述的结构参数和像素参数之后，即得到三维管状内壁片段Jn，构成三维管状内壁片段集{Jn}。
进一步地，在步骤S40中，首先将每个斜切圆柱筒体的侧壁进行变形处理，使得每个斜切圆柱筒体的顶端所在面调整为与相邻的下一个斜切圆柱筒体的底端所在面匹配，形成变形后的三维管状内壁片段。
图9所示为图8所述三维管状内壁片段Jn沿P1P2Q2Q1平面的截面图，其中an为当前帧原始消化道图像In时刻胶囊的轴线轴an+1为下一帧原始消化道图像In+1时刻胶囊的轴线轴，两图像帧具有旋转矩阵RT，n+1，对应姿态转换生成的轴线轴偏转角度θn+1。所述变形处理为将三维管状内壁片段Jn的顶端平面401变换为变形后的三维管状内壁片段Un的顶端平面402。所述两个平面的中心点O2保持不变。变形处理中，三维管状内壁片段侧壁的像素按照棱的伸缩进行拉伸与压缩。优选的，拉伸变形进行像素插值填充。
接着，将各个变形后的三维管状内壁片段的顶端和底端首尾连接，并且按照各个对应的姿态转换值对各个对各个变形后的三维管状内壁片段进行旋转，形成消化道三维全景图像。
如图10所示，选取三维管状内壁片段Un-1和Un，两者分别具有下底面中心On-1，1和On，1，以及上顶面中心On-1，2和On，2。拼接时，将Un下底面中心On，1与Un-1的上顶面中心On-1，2重合。将Un的轴线轴an相对Un-1的轴线轴an-1偏转θn角度，将Un的u轴相对Un-1的u轴偏转角度。以此类推完成所有{Un}的操作，生成消化道三维全景图像。
进一步地，在步骤S50中，当使用者在平移、旋转和缩放进行病灶识别与判断后，确定目标区域，当使用者点击目标区域时，可以确定目标区域所在的 三维管状内壁片段Un以及对应的原始消化道图像In。
在步骤S60中，建立空间坐标系，该空间坐标系为以第一帧原始消化道图像I1的中心作为原点的笛卡尔坐标系。计算三维管状内壁片段Un下底面中心On，1在空间坐标系中的坐标作为全局坐标：
O1，1＝[0 0 0]T，D1＝HD
进一步，则可根据图8所示三维管状内壁片段计算目标区域X的局部坐标。
目标区域X的局部极坐标为则目标区域X的局部笛卡尔坐标为
计算目标区域X的最终坐标为w＝On，1+RT，nwloc，继而获得目标区域X的位置信息。
图11示出了对原始消化道图像进行处理到最后定位到目标区域的整体过程。
本实施例二还公开了一种计算机可读存储介质，计算机可读存储介质存储有消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现上述的消化道的三维全景识别定位方法。
本实施例三还公开了一种计算机设备，在硬件层面，如图12所示，该计算机设备包括处理器12、内部总线13、网络接口14、计算机可读存储介质11。处理器12从计算机可读存储介质中读取对应的计算机程序然后运行，在逻辑层面上形成请求处理装置。当然，除了软件实现方式之外，本说明书一个或多个实施例并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。计算机可读存储介质11上存储有消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现上述的消化道的三维全景识别定位方法。
计算机可读存储介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机可读存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只 读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带、磁盘存储、量子存储器、基于石墨烯的存储介质或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。
上面对本发明的具体实施方式进行了详细描述，虽然已表示和描述了一些实施例，但本领域技术人员应该理解，在不脱离由权利要求及其等同物限定其范围的本发明的原理和精神的情况下，可以对这些实施例进行修改和完善，这些修改和完善也应在本发明的保护范围内。

Claims (17)

  1. 一种消化道的三维全景识别定位方法，其中，所述三维全景识别定位方法包括：
    获取胶囊内窥镜拍摄到的原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息；
    基于胶囊内窥镜的结构参数和帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像，构成新增区域图像集；
    构建三维管状内壁片段集，其中所述三维管状内壁片段集中每个片段根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到；
    将所述三维管状内壁片段集中的各个片段依序进行首尾拼接，形成消化道三维全景图像；
    在所述消化道三维全景图像中识别出目标区域后，确定目标区域所在的三维管状内壁片段；
    根据所述目标区域位于所在三维管状内壁片段的坐标、各个所述三维管状内壁片段的结构长度、帧旋转信息，确定所述目标区域的位置信息。
  2. 根据权利要求1所述的消化道的三维全景识别定位方法，其中，所述胶囊内窥镜上设置有姿态传感器，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
    获取姿态传感器采集到的姿态信息集合和姿态传感器的固定采样时间，其中所述姿态信息集合包括连续多个时刻的三轴方向上的旋转角速度；
    基于所述姿态信息集合和所述固定采样时间计算得到每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息。
  3. 根据权利要求1所述的消化道的三维全景识别定位方法，其中，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
    获取每一帧原始消化道图像与前一帧原始消化道图像进行特征匹配，获得若干对特征匹配点；
    基于各个所述特征匹配点的像素坐标值构建得到本质矩阵；
    基于所述本质矩阵分别得到帧旋转矩阵和平移矩阵的若干个估计值；
    根据若干对特征匹配点的其中一对特征匹配点的像素坐标值、帧旋转矩阵和平移矩阵的若干个估计值确定符合预设条件的帧旋转矩阵，作为帧旋转信息。
  4. 根据权利要求1所述的消化道的三维全景识别定位方法，其中，基于胶囊内窥镜的结构参数和对应帧的帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像的方法包括：
    根据胶囊内窥镜的结构参数确定每一帧原始消化道图像中的标志投影线；
    根据待截取的当前帧原始消化道图像的帧旋转信息计算得到当前帧原始消化道图像与相邻的前一帧原始消化道图像之间的姿态转换值；
    根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线；
    截取所述当前帧原始消化道图像中标志投影线和成像投影线之间的图像，作为新增区域图像。
  5. 根据权利要求4所述的消化道的三维全景识别定位方法，其中，根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线的方法包括：
    根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的若干候选投影线；
    计算前一帧原始消化道图像的标志投影线以及各所述候选投影线的离散量化像素灰度值序列；
    依次计算前一帧原始消化道图像的标志投影线的离散量化像素灰度值序列与各所述候选投影线的离散量化像素灰度值序列之间的莱文斯坦距离；
    将符合预设条件的莱文斯坦距离所对应的候选投影线作为成像投影线。
  6. 根据权利要求4所述的消化道的三维全景识别定位方法，其中，所述三维管状内壁片段为两端开放的斜切圆柱筒体，所述斜切圆柱筒体的顶端所在面为平面，所述斜切圆柱筒体的底端所在面为斜面。
  7. 根据权利要求6所述的消化道的三维全景识别定位方法，其中，根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到所述三维管状内壁片段的方法包括：
    根据所述胶囊内窥镜的结构参数、帧旋转信息计算得到所述斜切圆柱筒体 的直径、倾斜角度以及斜切圆柱筒体的长度；
    根据所述新增区域图像的像素值计算得到所述斜切圆柱筒体的内壁面的像素值。
  8. 根据权利要求7所述的消化道的三维全景识别定位方法，其中，将所述三维管状内壁片段集中的各个片段依序进行首尾拼接的方法包括：
    将每个斜切圆柱筒体的侧壁进行变形处理，使得每个斜切圆柱筒体的顶端所在面调整为与相邻的下一个斜切圆柱筒体的底端所在面匹配，形成变形后的三维管状内壁片段；
    将各个变形后的三维管状内壁片段的顶端和底端首尾连接，并且按照各个对应的姿态转换值对各个变形后的三维管状内壁片段进行旋转，形成消化道三维全景图像。
  9. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现权利要求1所述的消化道的三维全景识别定位方法。
  10. 根据权利要求9所述的计算机可读存储介质，其中，所述胶囊内窥镜上设置有姿态传感器，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
    获取姿态传感器采集到的姿态信息集合和姿态传感器的固定采样时间，其中所述姿态信息集合包括连续多个时刻的三轴方向上的旋转角速度；
    基于所述姿态信息集合和所述固定采样时间计算得到每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息。
  11. 根据权利要求9所述的计算机可读存储介质，其中，获取原始消化道图像集中每一帧原始消化道图像相对于相邻的前一帧原始消化道图像的帧旋转信息的方法包括：
    获取每一帧原始消化道图像与前一帧原始消化道图像进行特征匹配，获得若干对特征匹配点；
    基于各个所述特征匹配点的像素坐标值构建得到本质矩阵；
    基于所述本质矩阵分别得到帧旋转矩阵和平移矩阵的若干个估计值；
    根据若干对特征匹配点的其中一对特征匹配点的像素坐标值、帧旋转矩阵 和平移矩阵的若干个估计值确定符合预设条件的帧旋转矩阵，作为帧旋转信息。
  12. 根据权利要求9所述的计算机可读存储介质，其中，基于胶囊内窥镜的结构参数和对应帧的帧旋转信息在原始消化道图像集中的每一帧原始消化道图像上截取新增区域图像的方法包括：
    根据胶囊内窥镜的结构参数确定每一帧原始消化道图像中的标志投影线；
    根据待截取的当前帧原始消化道图像的帧旋转信息计算得到当前帧原始消化道图像与相邻的前一帧原始消化道图像之间的姿态转换值；
    根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线；
    截取所述当前帧原始消化道图像中标志投影线和成像投影线之间的图像，作为新增区域图像。
  13. 根据权利要求12所述的计算机可读存储介质，其中，根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的成像投影线的方法包括：
    根据胶囊内窥镜的结构参数和所述姿态转换值确定所述前一帧原始消化道图像的标志投影线在所述当前帧原始消化道图像中的若干候选投影线；
    计算前一帧原始消化道图像的标志投影线以及各所述候选投影线的离散量化像素灰度值序列；
    依次计算前一帧原始消化道图像的标志投影线的离散量化像素灰度值序列与各所述候选投影线的离散量化像素灰度值序列之间的莱文斯坦距离；
    将符合预设条件的莱文斯坦距离所对应的候选投影线作为成像投影线。
  14. 根据权利要求12所述的计算机可读存储介质，其中，所述三维管状内壁片段为两端开放的斜切圆柱筒体，所述斜切圆柱筒体的顶端所在面为平面，所述斜切圆柱筒体的底端所在面为斜面。
  15. 根据权利要求14所述的计算机可读存储介质，其中，根据胶囊内窥镜的结构参数、帧旋转信息和新增区域图像构建得到所述三维管状内壁片段的方法包括：
    根据所述胶囊内窥镜的结构参数、帧旋转信息计算得到所述斜切圆柱筒体的直径、倾斜角度以及斜切圆柱筒体的长度；
    根据所述新增区域图像的像素值计算得到所述斜切圆柱筒体的内壁面的像素值。
  16. 根据权利要求15所述的计算机可读存储介质，其中，将所述三维管状内壁片段集中的各个片段依序进行首尾拼接的方法包括：
    将每个斜切圆柱筒体的侧壁进行变形处理，使得每个斜切圆柱筒体的顶端所在面调整为与相邻的下一个斜切圆柱筒体的底端所在面匹配，形成变形后的三维管状内壁片段；
    将各个变形后的三维管状内壁片段的顶端和底端首尾连接，并且按照各个对应的姿态转换值对各个变形后的三维管状内壁片段进行旋转，形成消化道三维全景图像。
  17. 一种计算机设备，其中，所述计算机设备包括计算机可读存储介质、处理器和存储在所述计算机可读存储介质中的消化道的三维全景识别定位程序，所述消化道的三维全景识别定位程序被处理器执行时实现权利要求1所述的消化道的三维全景识别定位方法。
PCT/CN2023/097186 2022-06-22 2023-05-30 消化道的三维全景识别定位方法、存储介质和计算机设备 WO2023246441A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210713965.X 2022-06-22
CN202210713965.XA CN114782470B (zh) 2022-06-22 2022-06-22 消化道的三维全景识别定位方法、存储介质和设备

Publications (1)

Publication Number Publication Date
WO2023246441A1 true WO2023246441A1 (zh) 2023-12-28

Family

ID=82422432

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/097186 WO2023246441A1 (zh) 2022-06-22 2023-05-30 消化道的三维全景识别定位方法、存储介质和计算机设备

Country Status (2)

Country Link
CN (1) CN114782470B (zh)
WO (1) WO2023246441A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114782470B (zh) * 2022-06-22 2022-09-13 浙江鸿禾医疗科技有限责任公司 消化道的三维全景识别定位方法、存储介质和设备
CN118037963A (zh) * 2024-04-09 2024-05-14 广州思德医疗科技有限公司 消化腔内壁三维模型的重建方法、装置、设备和介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111145238A (zh) * 2019-12-12 2020-05-12 中国科学院深圳先进技术研究院 单目内窥镜图像的三维重建方法、装置及终端设备
WO2021141253A1 (ko) * 2020-01-10 2021-07-15 주식회사 인트로메딕 캡슐 내시경의 위치정보를 기반으로 캡슐 내시경의 위치를 파악하는 시스템 및 방법
CN113808253A (zh) * 2021-08-31 2021-12-17 武汉理工大学 场景三维重建的动态对象处理方法、系统、设备及介质
CN114066781A (zh) * 2022-01-18 2022-02-18 浙江鸿禾医疗科技有限责任公司 胶囊内窥镜肠道图像的识别定位方法、存储介质和设备
CN114782470A (zh) * 2022-06-22 2022-07-22 浙江鸿禾医疗科技有限责任公司 消化道的三维全景识别定位方法、存储介质和设备

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4631057B2 (ja) * 2004-02-18 2011-02-16 国立大学法人大阪大学 内視鏡システム
US9375132B2 (en) * 2011-06-23 2016-06-28 Kabushiki Kaisha Toshiba Medical image processing apparatus and medical image diagnosis apparatus
JP5863374B2 (ja) * 2011-10-11 2016-02-16 本田技研工業株式会社 画像処理方法
CN103412401B (zh) * 2013-06-07 2015-05-13 中国科学院上海光学精密机械研究所 内窥镜和管道内壁三维图像重建方法
CN110769731B (zh) * 2017-06-15 2022-02-25 奥林巴斯株式会社 内窥镜系统、内窥镜用处理系统、图像处理方法
WO2020114806A1 (en) * 2018-12-06 2020-06-11 Koninklijke Philips N.V. 3-d virtual endoscopy rendering
CN113544743B (zh) * 2019-09-20 2024-03-12 Hoya株式会社 内窥镜用处理器、程序、信息处理方法和信息处理装置
WO2021064867A1 (ja) * 2019-10-01 2021-04-08 日本電気株式会社 画像処理装置、制御方法及び記憶媒体
CN113610887A (zh) * 2021-05-26 2021-11-05 江苏势通生物科技有限公司 胶囊内窥镜运动拍摄路径的确定方法、存储介质和设备
CN113538335A (zh) * 2021-06-09 2021-10-22 香港中文大学深圳研究院 一种无线胶囊内窥镜的体内相对定位方法和装置
CN114022547A (zh) * 2021-09-15 2022-02-08 苏州中科华影健康科技有限公司 一种内窥镜图像检测方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111145238A (zh) * 2019-12-12 2020-05-12 中国科学院深圳先进技术研究院 单目内窥镜图像的三维重建方法、装置及终端设备
WO2021141253A1 (ko) * 2020-01-10 2021-07-15 주식회사 인트로메딕 캡슐 내시경의 위치정보를 기반으로 캡슐 내시경의 위치를 파악하는 시스템 및 방법
CN113808253A (zh) * 2021-08-31 2021-12-17 武汉理工大学 场景三维重建的动态对象处理方法、系统、设备及介质
CN114066781A (zh) * 2022-01-18 2022-02-18 浙江鸿禾医疗科技有限责任公司 胶囊内窥镜肠道图像的识别定位方法、存储介质和设备
CN114782470A (zh) * 2022-06-22 2022-07-22 浙江鸿禾医疗科技有限责任公司 消化道的三维全景识别定位方法、存储介质和设备

Also Published As

Publication number Publication date
CN114782470B (zh) 2022-09-13
CN114782470A (zh) 2022-07-22

Similar Documents

Publication Publication Date Title
WO2023246441A1 (zh) 消化道的三维全景识别定位方法、存储介质和计算机设备
WO2023138544A1 (zh) 胶囊内窥镜肠道图像的识别定位方法、存储介质和设备
Bao et al. A video-based speed estimation technique for localizing the wireless capsule endoscope inside gastrointestinal tract
KR102237441B1 (ko) 뉴럴 네트워크를 이용하여 캡슐 내시경 영상으로부터 병변 판독 방법 및 장치
US7922652B2 (en) Endoscope system
Bao et al. A computer vision based speed estimation technique for localiz ing the wireless capsule endoscope inside small intestine
CN105957007A (zh) 基于特征点平面相似度的图像拼接方法
CN111091562B (zh) 一种消化道病灶大小测量方法及系统
US20220012457A1 (en) Image processing method, microscope, image processing system, and medium based on artificial intelligence
JP7211621B2 (ja) 画像生成装置、および画像生成プログラム
EP1889204A2 (en) A fast 2d-3d image registration method with application to continuously guided endoscopy
CN104463778A (zh) 一种全景图生成方法
JP6501800B2 (ja) 信頼度マッチング付き生体内マルチカメラカプセルからの画像の再構築
CN106157246A (zh) 一种全自动的快速柱面全景图像拼接方法
CN108090954A (zh) 基于图像特征的腹腔环境地图重建与腹腔镜定位的方法
CN114022547A (zh) 一种内窥镜图像检测方法、装置、设备及存储介质
CN115082617A (zh) 基于多视图优化的管道三维重建方法、装置及存储介质
CN112637519A (zh) 多路4k准实时拼接视频的全景拼接算法
CN111491157B (zh) 一种构建手机3d模型引导图像全面采集的方法
KR100930594B1 (ko) 안면 영상 촬영장치 및 그의 안면 특징점 검출 방법
CN112633113A (zh) 跨摄像头的人脸活体检测方法及系统
CN116687328A (zh) 导管的移动控制装置、方法及存储介质
CN112581460B (zh) 扫描规划方法、装置、计算机设备和存储介质
Safavian et al. Endoscopic measurement of the size of gastrointestinal polyps using an electromagnetic tracking system and computer vision-based algorithm
CN116469101A (zh) 数据标注方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23826090

Country of ref document: EP

Kind code of ref document: A1