CN117204791A - Endoscopic instrument guidance method and system - Google Patents

Abstract

The invention discloses an endoscopic instrument guidance method and system, which belongs to the field of medical image processing. By performing feature extraction and matching on the collected endoscopic image frame sequence, the real-time position of the camera is obtained through calculation; and then the real-time position of the camera is obtained during the doctor's operation. When the instrument is installed, the angle and extension distance of the instrument are obtained through the sensor, and the relative position relationship between the camera and the instrument is calculated; finally, the overlapping portion of the field of view of the instrument and the camera is obtained in the three-dimensional space, and projected to the corresponding camera screen to achieve real-time guidance. Through the above steps , only uses two additional sensors to obtain the extended position of the endoscopic instrument, achieving real-time guidance. It is small in size and does not add additional burden to the equipment. The system has fast processing speed, good real-time performance, and is compatible with different image resolutions from 480p to 1080p. , which can support up to 60fps frame rate output. The system has high position detection accuracy and good consistency. It can maintain high instrument positioning and guidance accuracy during long-term inspections/surgeries (more than 2 hours).

Description

Endoscopic instrument guidance method and system

Technical field

The present invention relates to the field of medical image processing, and in particular to endoscopic instrument guidance methods and systems.

Background technique

Medical endoscope is a minimally invasive inspection or surgical auxiliary medical device. An endoscope includes an optical lens, a camera and a mechanical auxiliary device. The endoscope enters the human body through the natural orifices of the human body or small external incisions. It can observe internal diseased tissues and assist in completing operations such as diagnosis, treatment and surgery. Compared with traditional medical methods, endoscopic systems have significant advantages such as less trauma, ease of use, and shorter operation time, and are widely used in various departments.

The endoscope enters the human body through a natural orifice. The insertion part at the front end of the endoscope body is a soft and flexible structure, including a camera, a light source, a water and vapor channel, and an instrument channel; the rear end of the endoscope body is an operating handle, which is connected to the host computer. When the doctor uses an endoscope to perform biopsy or surgery, he controls the direction of the front end of the scope through the handle. After the scope is fixed, he inserts the surgical instrument into the instrument hole and controls the extension angle of the instrument by operating the forceps lifting wire inside the hole. Complete the operation. Since the endoscope camera and the exit of the instrument channel are not on the same plane, there are often situations where the instrument is out of the camera's field of view. At this time, the doctor cannot see the instrument in the screen and needs to rely on experience to adjust the instrument to the appropriate angle, which is inconvenient to operate.

Currently, commonly used endoscopic surgery/examination guidance methods mostly require the assistance of external large-scale equipment, such as imaging equipment such as CT, ultrasound, and MRI. Intraoperative guidance is carried out through multi-mode image registration or three-dimensional modeling, which requires a large amount of calculation and is large-scale. The equipment takes up a lot of space and is expensive.

Contents of the invention

In order to overcome the shortcomings of the existing technology, one of the purposes of the present invention is to provide an endoscopic instrument guidance method to realize instrument guidance during surgery/inspection without the need for large-scale equipment, with small computational load, high processing speed, and good real-time performance. Strong scalability.

In order to overcome the shortcomings of the existing technology, the second object of the present invention is to provide an endoscopic instrument guidance method to implement an instrument system during surgery/inspection without the need for large-scale equipment, with low computational load, high processing speed, and good real-time performance. Strong scalability.

One of the purposes of the present invention is achieved by adopting the following technical solutions:

An endoscopic instrument guidance method includes the following steps:

S1: The camera acquires multiple endoscopic images, performs feature extraction and matching on the acquired adjacent frame images, and solves the camera pose based on the matching results. After acquiring the camera pose, obtain the spatial information of all feature points. According to Spatial information optimizes camera pose;

S2: When operating an instrument, the sensor is used to obtain the angle and distance the instrument moves relative to the instrument channel, thereby obtaining the position of the instrument head relative to the instrument channel, and calculating the relative positional relationship between the camera and the instrument;

S3: Connect the instrument head end and the instrument channel plane in the three-dimensional space, construct an extension line, extend the instrument into the field of view of the camera, and project the extension line onto the camera image to predict and guide the position of the instrument.

Further, steps S1 and S2 operate in parallel and step S1 runs through the entire endoscopic instrument guidance process.

Further, step S1 specifically includes the following steps:

S11: Camera initialization: rotate and translate the initial lens to obtain two frames of initial images;

S12: Feature extraction: extract feature points in two adjacent frames of images and calculate the BRIEF descriptor;

S13: Perform descriptor matching on all feature points and obtain feature point pairs;

S14: Camera motion solution: When the number of feature point pairs is less than the preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, solve the camera motion based on the feature point pairs to obtain the camera rotation matrix R and Translation matrix t, get the latest position of the camera;

S15: Obtain the spatial information of the feature points: through triangulation, use different 2D projections of the same spatial feature points in the two frames before and after, and obtain the spatial position of the feature points through the least squares method;

S16: Camera pose optimization: Store the obtained three-dimensional space feature point information. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, the camera pose is optimized;

S17: Continue to acquire images and repeat steps S12 to S16 until the endoscopic instrument guidance is completed.

Further, in step S11, the camera coordinate system when collecting the first frame image is used as the world coordinate system and used as the coordinate system calculated in the endoscopic instrument guidance method.

Further, in step S14, the default value is 8, and 2D-2D epipolar constraints are used to solve the camera motion.

Further, step S2 specifically includes the following steps:

S21: Align the instrument with the initial calculation plane: operate the instrument into the instrument channel of the endoscope, read the position sensor reading during insertion, stop the operation when the reading reaches the internal length of the endoscopic instrument channel, and reset the position sensor reading to zero , ensure that the head end of the instrument is aligned with the extending plane of the channel;

S22: Operate the instrument and obtain motion parameters: Operate the instrument and obtain the distance and deflection angle of the instrument relative to the initial position by reading the readings of the position sensor and the rotary encoder respectively;

S23: Calculation of the position of the instrument head end: Since the deflection of the instrument is a single degree of freedom motion, the position relationship of the instrument head end relative to the instrument channel plane is calculated based on the movement distance and deflection angle;

S24: Determine the positional relationship between the instrument and the camera: The positional relationship between the instrument channel plane and the camera is fixed and known. The positional relationship between the instrument and the camera is obtained through the positional relationship between the instrument head end and the instrument channel plane.

Further, in step S23, the positional relationship of the instrument head end relative to the instrument channel plane is the rotation matrix R and the translation vector t.

Further, step S3 specifically includes the following steps:

S31: Find the nearest and farthest feature points: read the camera position at the current moment, obtain the set of feature points extracted during camera positioning, find the distances between all feature points and the camera, and obtain the farthest point and the closest point from the camera respectively. ;

S32: Obtain the camera frustum:

S33: Find the intersection point of the extension line of the instrument: connect the head end of the instrument and the end plane point, draw the ray to intersect with the camera's visual frustum, obtain two intersection points, and solve for the spatial position coordinates of the two intersection points;

S34: Construct a guide line: According to the camera projection equation, project the three-dimensional coordinates of the two intersection points onto the pixel plane of the camera to obtain the two-dimensional pixel coordinates. Connect the two intersection points in the camera imaging screen to obtain the instrument guide line to achieve guidance.

Further, step S32 specifically includes the following steps:

S321: Connect the camera point to the nearest and farthest points respectively, and take the planes perpendicular to the connection between the nearest and farthest points as the nearest and farthest frustum planes;

S322: Calculate the angle θ between the closest viewing cone plane and the farthest viewing cone plane, and reversely rotate the two planes at the same time Obtain mutually parallel viewing frustum planes;

S323: Extract rays according to the horizontal and vertical field of view angles of the camera, intersect with the nearest/farthest view cone plane, and obtain the near clipping plane and the far clipping plane. The middle part surrounded by the two clipping planes is the visible view cone space of the camera.

The second object of the present invention is achieved by adopting the following technical solutions:

An endoscopic instrument guidance system used to implement the above endoscopic instrument guidance method, including a camera positioning module, an instrument position detection module and an instrument guidance module. The camera positioning module calculates the camera pose according to the images collected by the camera. And optimize the camera posture; the instrument position detection module includes an encoder and a position sensor. The encoder obtains the angle of movement of the instrument relative to the instrument channel. The position sensor obtains the distance of movement of the instrument relative to the instrument channel. Through the angle and The distance calculates the position of the instrument tip relative to the instrument channel; the instrument guidance module connects the instrument head end and the channel plane in a three-dimensional space, constructs an extension line, extends the instrument into the field of view of the camera, and projects the extension line to the camera image Prediction and guidance of device position are achieved.

Compared with the existing technology, the endoscopic instrument guidance method of the present invention performs feature extraction and matching on the collected endoscopic image frame sequence, and obtains the real-time position of the camera through calculation; then, when the doctor operates the instrument, the instrument's position is obtained through the sensor. angle and protrusion distance to calculate the relative positional relationship between the camera and the instrument; finally, the overlapping portion of the field of view of the instrument and the camera is obtained in the three-dimensional space, and projected to the corresponding camera image to achieve real-time guidance. Through the above steps, it is obtained only through two additional sensors. The extended position of endoscopic instruments enables real-time guidance. It is small in size and does not add additional burden to the equipment. The system has fast processing speed and good real-time performance. It is compatible with different image resolutions from 480p to 1080p, and can support up to 60fps frame rate output. , the system has high position detection accuracy and good consistency, and can maintain high instrument positioning and guidance accuracy during long-term inspections/operations (more than 2 hours).

Description of the drawings

Figure 1 is a flow chart of the endoscopic instrument guidance method of the present invention;

Figure 2 is a flow chart for optimizing camera posture in the endoscopic instrument guidance method of Figure 1;

Figure 3 is a flow chart of instrument position detection and instrument guidance in the endoscopic instrument guidance method of Figure 1;

Figure 4 is a schematic diagram of the front end structure of the endoscope according to the present invention;

Figure 5 is a schematic diagram of instrument guidance of the endoscopic instrument guidance method of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

It should be noted that when a component is referred to as being "fixed to" another component, it can be directly on the other component or another intermediate component may be present through which it is fixed. When a component is said to be "connected" to another component, it can be directly connected to the other component or there may be another intermediate component present at the same time. When a component is said to be "disposed on" another component, it can be directly located on the other component or another intervening component may be present. The terms "vertical," "horizontal," "left," "right" and similar expressions are used herein for illustrative purposes only.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which the invention belongs. The terminology used herein in the description of the invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Figure 1 shows the endoscopic instrument guidance method of the present invention, which includes the following steps:

S1: The camera acquires multiple endoscopic images, performs feature extraction and matching on the acquired adjacent frame images, and solves the camera pose based on the matching results. After acquiring the camera pose, obtain the spatial information of all feature points. According to Spatial information optimizes camera pose;

S2: When operating an instrument, the sensor is used to obtain the angle and distance the instrument moves relative to the instrument channel, thereby obtaining the position of the instrument head relative to the instrument channel, and calculating the relative positional relationship between the camera and the instrument;

S3: Connect the instrument head end and the instrument channel plane in the three-dimensional space, construct an extension line, extend the instrument into the field of view of the camera, and project the extension line onto the camera image to predict and guide the position of the instrument.

Steps S1 and S2 operate in parallel and step S1 runs through the entire endoscopic instrument guidance process to realize real-time guidance and correction of the endoscopic instrument.

Please continue to refer to Figure 2. Step S1 specifically includes the following steps:

S11: Camera initialization: Rotate and translate the initial lens, obtain two frames of initial images, and use the camera coordinate system when collecting the first frame of image as the world coordinate system and the coordinate system calculated in the endoscopic instrument guidance method;

S12: Feature extraction: Use the ORB algorithm to extract feature points in two adjacent frames of images and calculate the BRIEF descriptor;

S13: Use the fast approximate nearest neighbor algorithm (FLANN) to perform descriptor matching on all feature points and obtain feature point pairs;

S14: Camera motion solution: When the number of feature point pairs is less than the preset value, return to step S11. When the number of feature point pairs is greater than or equal to the preset value, use 2D-2D epipolar constraints to solve the camera motion based on the feature point pairs. , obtain the camera rotation matrix R and translation matrix t, and obtain the latest position of the camera;

S15: Obtain the spatial information of the feature points: through triangulation, use different 2D projections of the same spatial feature points in the two frames before and after, and obtain the spatial position of the feature points through the least squares method;

S16: Camera pose optimization: Store the obtained three-dimensional space feature point information. When the number of stored image frames is less than 10, the camera pose remains unchanged; when the number of stored image frames is greater than or equal to 10, use the BA cost function to perform camera positioning. posture optimization;

S17: Continue to acquire images and repeat steps S12 to S16 until the endoscopic instrument guidance is completed.

Specifically, in step S14, the default value is 8, and 2D-2D epipolar constraints are used to solve the camera motion.

Please continue to refer to Figure 3 and Figure 4. Step S2 specifically includes the following steps:

S21: Align the instrument with the initial calculation plane: operate the instrument into the instrument channel of the endoscope, read the position sensor reading during insertion, stop the operation when the reading reaches the internal length of the endoscopic instrument channel, and reset the position sensor reading to zero , ensure that the head end of the instrument is aligned with the extending plane of the channel;

S22: Operate the instrument and obtain motion parameters: Operate the instrument and obtain the distance and deflection angle of the instrument relative to the initial position by reading the readings of the position sensor and the rotary encoder respectively;

S23: Calculation of the position of the instrument head end: Since the deflection of the instrument is a single degree of freedom motion, the position relationship of the instrument head end relative to the instrument channel plane is calculated based on the movement distance and deflection angle;

S24: Determine the positional relationship between the instrument and the camera: The positional relationship between the instrument channel plane and the camera is fixed and known. The positional relationship between the instrument and the camera is obtained through the positional relationship between the instrument head end and the instrument channel plane.

Specifically, in step S23, the positional relationship between the instrument head end and the instrument channel plane is the rotation matrix R and the translation vector t.

Please continue to refer to Figure 5. Step S3 specifically includes the following steps:

S31: Find the nearest and farthest feature points: read the camera position at the current moment, obtain the set of feature points extracted during camera positioning, find the distances between all feature points and the camera, and obtain the farthest point and the closest point from the camera respectively. ;

S32: Obtain the camera frustum:

S33: Find the intersection point of the extension line of the instrument: connect the head end of the instrument and the end plane point, draw the ray to intersect with the camera's visual frustum, obtain two intersection points, and solve for the spatial position coordinates of the two intersection points;

S34: Construct a guide line: According to the camera projection equation, project the three-dimensional coordinates of the two intersection points onto the pixel plane of the camera to obtain the two-dimensional pixel coordinates. Connect the two intersection points in the camera imaging screen to obtain the instrument guide line to achieve guidance.

Specifically, step S32 specifically includes the following steps:

S321: Connect the camera point to the nearest and farthest points respectively, and take the planes perpendicular to the connection between the nearest and farthest points as the nearest and farthest frustum planes;

S322: Calculate the angle θ between the closest viewing cone plane and the farthest viewing cone plane, and reversely rotate the two planes at the same time Obtain mutually parallel viewing frustum planes;

S323: Extract rays according to the horizontal and vertical field of view angles of the camera, intersect with the nearest/farthest view cone plane, and obtain the near clipping plane and the far clipping plane. The middle part surrounded by the two clipping planes is the visible view cone space of the camera.

The present invention also relates to an endoscopic instrument guidance system for implementing the above endoscopic instrument guidance method, including a camera positioning module, an instrument position detection module and an instrument guidance module. The camera positioning module is based on the image collected by the camera. Calculate the camera pose and optimize the camera pose; the instrument position detection module includes an encoder and a position sensor, the encoder obtains the angle of movement of the instrument relative to the instrument channel, and the position sensor obtains the distance of movement of the instrument relative to the instrument channel , calculate the position of the instrument tip relative to the instrument channel through angle and distance; the instrument guidance module connects the instrument head end and the channel plane in a three-dimensional space, constructs an extension line, extends the instrument into the field of view of the camera, and extends the extension line Projection onto the camera image enables prediction and guidance of instrument position.

The endoscopic instrument guidance method of the present invention performs feature extraction and matching on the collected endoscopic image frame sequence, and obtains the real-time position of the camera through calculation; then when the doctor operates the instrument, the angle and extension distance of the instrument are obtained through the sensor, and the calculation The relative positional relationship between the camera and the instrument is obtained; finally, the overlapping portion of the field of view of the instrument and the camera is obtained in the three-dimensional space, and projected to the corresponding camera image to achieve real-time guidance. Through the above steps, the extension of the endoscopic instrument is obtained only through two additional sensors. position, realizing real-time guidance, small size, not adding extra burden to the device, fast system processing speed, good real-time performance, compatible with different image resolutions from 480p to 1080p, up to 60fps frame rate output, and high system position detection accuracy , has good consistency, and can maintain high instrument positioning and guidance accuracy during long-term inspections/operations (more than 2 hours).

The above embodiments only express several embodiments of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications to the above embodiments based on the essential technology of the present invention. and evolution, these all belong to the protection scope of the present invention.

Claims (10)

1. An endoscopic instrument guide method, comprising the steps of:

s1: the camera acquires a plurality of endoscope images, performs feature extraction and matching on the acquired adjacent frame images respectively, solves the pose of the camera according to the matching result, solves the spatial information of all feature points after acquiring the pose of the camera, and optimizes the pose of the camera according to the spatial information;

s2: when the instrument is operated, the angle and the distance of movement of the instrument relative to the instrument channel are obtained through the sensor, so that the position of the head end of the instrument relative to the instrument channel is obtained, and the relative position relation between the camera and the instrument is calculated;

s3: and connecting the instrument head end and the instrument channel plane in the three-dimensional space, constructing an extension line, extending the instrument to the field of view of the camera, and projecting the extension line to the camera image to realize the prediction and the guidance of the instrument position.

2. The endoscopic instrument guide method according to claim 1, wherein: steps S1 and S2 operate in parallel and step S1 extends through the entire endoscopic instrument guide procedure.

3. The endoscopic instrument guide method according to claim 1, wherein: the step S1 specifically comprises the following steps:

s11: camera initialization: rotating and translating the initial lens to obtain two frames of initial images;

s12: feature extraction: extracting feature points in two adjacent frames of images, and calculating BRIEF descriptors;

s13: carrying out descriptor matching on all the characteristic points to obtain characteristic point pairs;

s14: camera motion solution: returning to the step S11 when the number of the characteristic point pairs is smaller than a preset value, and solving the movement of the camera according to the characteristic point pairs to obtain a camera rotation matrix R and a translation matrix t when the number of the characteristic point pairs is larger than or equal to the preset value to obtain the latest position of the camera;

s15: acquiring spatial information of feature points: by means of triangulation, different 2D projections of the same spatial feature points in the front frame and the rear frame are utilized, and the spatial positions of the feature points are obtained through a least square method;

s16: and (3) optimizing the pose of the camera: storing the obtained three-dimensional space feature point information, and keeping the pose of the camera unchanged when the number of stored image frames is smaller than 10; when the stored image frame number is more than or equal to 10, performing pose optimization of the camera;

s17: continuing to acquire images, repeating steps S12 to S16 until the guiding of the endoscopic instrument is completed.

4. The endoscopic instrument guide method according to claim 3, wherein: in step S11, the camera coordinate system in which the first frame image is acquired is used as the world coordinate system, and is used as the coordinate system calculated in the endoscopic instrument guidance method.

5. The endoscopic instrument guide method according to claim 3, wherein: in step S14, the preset value is 8, and the 2D-2D epipolar constraint is adopted to solve the motion of the camera.

6. The endoscopic instrument guide method according to claim 1, wherein: the step S2 specifically comprises the following steps:

s21: alignment of the instrument with the initial calculation plane: operating the instrument to enter the instrument channel of the endoscope, reading the reading of the position sensor during insertion, stopping the operation when the reading reaches the inner length of the instrument channel of the endoscope, and zeroing the reading of the position sensor to ensure that the head end of the instrument is aligned with the extending plane of the channel;

s22: operating the instrument to obtain motion parameters: operating the instrument, and respectively obtaining the moving distance and the deflection angle of the instrument relative to the initial position through reading the readings of the position sensor and the rotary encoder;

s23: calculating the position of the instrument head end: because the deflection of the instrument is single-degree-of-freedom motion, the position relation of the head end of the instrument relative to the plane of the instrument channel is obtained according to the moving distance and the deflection angle;

s24: obtaining the position relation between the instrument and the camera: the positional relationship between the instrument channel plane and the camera is fixedly known, and the positional relationship between the instrument and the camera is obtained through the positional relationship between the instrument head end and the instrument channel plane.

7. The endoscopic instrument guide method according to claim 6, wherein: in step S23, the positional relationship of the instrument tip with respect to the instrument channel plane is the rotation matrix R and the translation vector t.

8. The endoscopic instrument guide method according to claim 1, wherein: the step S3 specifically comprises the following steps:

s31: the nearest and farthest feature points are obtained: reading the camera position at the current moment, acquiring a feature point set extracted from camera positioning, solving the distances between all feature points and the camera, and respectively acquiring the farthest point and the nearest point from the camera;

s32: acquiring a camera vision cone:

s33: solving the intersection point of the extension lines of the instrument: connecting the instrument head end with the tail end plane point, leading out rays to intersect with the camera view cone, obtaining two intersection points, and solving the spatial position coordinates of the two intersection points;

s34: constructing a guide wire: according to a camera projection equation, the three-dimensional coordinates of the two intersection points are projected to a pixel plane of a camera to obtain two-dimensional pixel coordinates, the two intersection points are connected in an imaging picture of the camera to obtain instrument guide lines, and guidance is achieved.

9. The endoscopic instrument guide method according to claim 8, wherein: the step S32 specifically includes the following steps:

s321: connecting the camera points with the nearest and farthest points respectively, and taking the surfaces of the nearest and farthest points perpendicular to the connecting lines as nearest and farthest viewing cone planes;

s322: calculating the included angle theta between the plane of the shortest sight cone and the plane of the farthest sight cone, and reversely rotating the two planes simultaneouslyObtaining mutually parallel viewing cone planes;

s323: and (3) extracting rays according to the transverse and longitudinal field angles of the camera, intersecting with the nearest/far view cone plane to obtain a near clipping surface and a far clipping surface, wherein the middle part surrounded by the two clipping surfaces is the visible cone space of the camera.

10. An endoscopic instrument guide system for performing the endoscopic instrument guide method of any of claims 1-9, wherein: the camera positioning module calculates the pose of the camera according to the image acquired by the camera and optimizes the pose of the camera; the instrument position detection module comprises an encoder and a position sensor, wherein the encoder obtains the movement angle of the instrument relative to the instrument channel, the position sensor obtains the movement distance of the instrument relative to the instrument channel, and the position of the tip of the instrument relative to the instrument channel is calculated through the angle and the distance; the instrument guiding module is connected with the instrument head end and the channel plane in a three-dimensional space, an extension line is constructed, the instrument is prolonged to be in the field of view of the camera, and the extension line is projected onto the camera image to realize the prediction and guiding of the instrument position.