CN114081420A - Accessory for endoscope, endoscope system, control method and device - Google Patents
Accessory for endoscope, endoscope system, control method and device Download PDFInfo
- Publication number
- CN114081420A CN114081420A CN202111501595.5A CN202111501595A CN114081420A CN 114081420 A CN114081420 A CN 114081420A CN 202111501595 A CN202111501595 A CN 202111501595A CN 114081420 A CN114081420 A CN 114081420A
- Authority
- CN
- China
- Prior art keywords
- lens group
- endoscope
- liquid lens
- group
- driver
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 239000007788 liquid Substances 0.000 claims abstract description 149
- 238000003384 imaging method Methods 0.000 claims abstract description 17
- 238000003780 insertion Methods 0.000 claims abstract description 10
- 230000037431 insertion Effects 0.000 claims abstract description 10
- 239000013598 vector Substances 0.000 claims description 36
- 238000013519 translation Methods 0.000 claims description 22
- 238000001514 detection method Methods 0.000 claims description 20
- 230000004927 fusion Effects 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 10
- 230000004075 alteration Effects 0.000 claims description 8
- 238000003708 edge detection Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 10
- 230000000007 visual effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000000354 decomposition reaction Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002789 length control Methods 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00131—Accessories for endoscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/002—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor having rod-lens arrangements
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Biomedical Technology (AREA)
- Optics & Photonics (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Biophysics (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Endoscopes (AREA)
Abstract
The invention provides an accessory for an endoscope, an endoscope system, a control method and a device, wherein the accessory comprises: the device comprises a tube assembly, a liquid lens group, a wedge-shaped lens group and a driver group; the proximal side of the tube assembly is for detachable connection with an insertion end of the endoscope; the wedge-shaped lens group is arranged on the far end side of the tube assembly, the liquid lens group and the wedge-shaped lens are arranged at intervals, and the driver group is electrically connected with the liquid lens group; the driver group is used for adjusting the curvature of the liquid lens group so as to adjust the focal length and the magnification of the endoscope, and the wedge-shaped lens group is used for increasing the imaging angle of the endoscope. Through the scope that sets up through wedge group also can effectively enlarge the art field in endoscope insertion end front side, through liquid lens and the driver group that corresponds, can realize focusing fast under the condition that the magnification freely switches over, guarantee to the regional clear continuous observation of art field.
Description
Technical Field
The invention relates to the technical field of electronics, in particular to an accessory for an endoscope, an endoscope system, a control method and a control device.
Background
With the continuous development of medical technology, endoscopes are widely used in modern medicine.
Although the endoscope in the prior art can directly observe the surgical field, the endoscope cannot provide variable magnification and variable visual field size during the use process, so that a doctor needs to continuously move the endoscope according to the region of interest.
Therefore, how to implement variable magnification while enlarging the field of view of the endoscope has become a problem to be solved in the art.
Disclosure of Invention
The invention provides an accessory for an endoscope, an endoscope system, a control method and a control device, which are used for solving the defect that the endoscope in the prior art cannot well realize quick focusing in the using process.
The present invention provides an accessory for an endoscope, the accessory comprising: the device comprises a tube assembly, a liquid lens group, a wedge-shaped lens group and a driver group;
the proximal side of the tube assembly is for detachable connection with an insertion end of the endoscope;
the wedge-shaped lens group is arranged on the far end side of the tube assembly, the liquid lens group and the wedge-shaped lens are arranged at intervals, and the driver group is electrically connected with the liquid lens group;
the driver group is used for adjusting the curvature of the liquid lens group so as to adjust the focal length and the magnification of the endoscope, and the wedge-shaped lens group is used for increasing the imaging angle of the endoscope.
According to an accessory for an endoscope provided by the present invention, the liquid lens group comprises a first liquid lens and a second liquid lens, the driver group comprises a first driver and a second driver;
the first driver is electrically connected with the first liquid lens, and the second driver is electrically connected with the second liquid lens;
the first liquid lens is used for focal length adjustment, and the second liquid lens is used for magnification adjustment.
According to the present invention there is provided an accessory for an endoscope, the accessory further comprising: a micro-motor;
the micro motor is connected with the wedge-shaped mirror group and used for controlling the wedge-shaped mirror group to adjust the inclination angle.
According to the present invention there is provided an accessory for an endoscope, the accessory further comprising: a concave lens group and a convex lens group;
the concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group;
the concave lens group is used for eliminating chromatic aberration, and the convex lens group is used for focusing light.
According to the present invention there is provided an accessory for an endoscope, the accessory further comprising: the first input unit and the second input unit are connected with the liquid lens group;
wherein the first input unit is configured to adjust a curvature of the liquid lens group in response to an increase magnification input to magnify an image of the endoscope;
the second input unit is used for responding to the input of the reduction magnification, and adjusting the curvature of the liquid lens group so as to reduce the imaging of the endoscope.
Optionally, a third liquid lens is further included in the liquid lens group, and a third driver is further included in the driver group;
the third driver is electrically connected with the third liquid lens;
the third liquid lens and the first liquid lens are commonly used for focal length adjustment.
The present invention also provides an endoscope system comprising: an endoscope and an accessory for an endoscope as described above.
The invention also provides a control method based on the endoscope system, which comprises the following steps:
performing definition detection on an edge region imaged by an endoscope;
under the condition that the definition of the pixel points in the edge area is lower than a preset threshold, searching and approaching the driving voltage of the driver set in a preset voltage interval by a golden ratio division method to obtain a target driving voltage of the driver set, wherein the target driving voltage is the driving voltage corresponding to the pixel points in the edge area when the definition is the highest;
and adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
According to a control method based on the endoscope system, the method for detecting the definition of the edge area imaged by the endoscope comprises the following steps:
under the condition that the endoscope detects that the object distance changes, performing definition detection on an edge area imaged by the endoscope;
or; under the condition that the magnification of the liquid lens group is changed, the definition detection is carried out on the edge area imaged by the endoscope.
Optionally, after the adjusting the curvature of the liquid lens group based on the target driving voltage to image according to the adjusted liquid lens group, the method further includes:
acquiring a plurality of groups of image groups under different wedge-shaped mirror angles, which are acquired in the process of searching and approaching the driving voltage of the driver group in a preset voltage interval, wherein each group of image groups comprises at least two images under different magnifications;
SIFT feature point detection and matching are carried out on each picture, and translation vectors and rotation vectors among the images are determined;
and splicing each group of image groups based on the translation vector and the rotation direction to obtain a target image.
Optionally, after the step of determining a translation vector and a rotation vector between the images, further comprising;
constructing a first depth map based on the translation vector and the rotation vector;
performing focusing edge detection on each image by using a Laplace Gaussian kernel to obtain a second depth map, wherein the corresponding depth value of each pixel in the second depth map is determined based on a layer where the norm maximum value of the Laplace Gaussian kernel is located;
normalizing the second depth map, and performing depth optimization processing on the second depth map to obtain a third depth map;
adding the first depth map and the third depth map, and averaging to obtain a fourth fused depth map;
splicing the fourth fusion depth map based on the translation vector and the rotation vector to obtain a target fusion map;
and determining a target three-dimensional reconstruction model based on the target fusion depth map and three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on the adjusted imaging of the liquid lens group.
The present invention also provides a control apparatus comprising:
the definition measuring module is used for carrying out definition detection on the edge area imaged by the endoscope;
the extreme value searching module is used for searching and approaching the driving voltage of the driver set in a preset voltage interval by comparing the contrast of the current image with the contrast of the previous frame of image and using a golden ratio segmentation method to obtain the target driving voltage of the driver set, wherein the target driving voltage is the driving voltage corresponding to the highest definition of the pixel points in the edge area;
and the adjusting module is used for adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and operable on the processor, wherein the processor implements the steps of any of the above control methods when executing the program.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the control method as described in any one of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, carries out the steps of the control method according to any one of the above.
According to the accessory, the endoscope system, the control method and the device for the endoscope, the range of the surgical field can be effectively expanded through the wedge-shaped lens group arranged on the front side of the insertion end of the endoscope, and rapid focusing can be realized under the condition that the magnification is freely switched through the liquid lens and the corresponding driver group, so that clear and continuous observation of the surgical field area is ensured.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic structural view of an attachment for an endoscope provided in an embodiment of the present application;
FIG. 2 is a schematic structural diagram of an endoscope system provided by an embodiment of the present application;
FIG. 3 is a second schematic structural view of an endoscope system as described in the embodiments of the present application;
fig. 4 is a schematic flowchart of a control method provided in an embodiment of the present application;
fig. 5 is a schematic structural diagram of a control device according to an embodiment of the present application;
fig. 6 illustrates a physical structure diagram of an electronic device.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic structural view of an accessory for an endoscope provided in an embodiment of the present application, as shown in fig. 1, including: the device comprises a tube assembly 1, a liquid lens group 2, a wedge-shaped lens group 3 and a driver group 4;
the proximal side 11 of the tube assembly 1 is intended for detachable connection with the insertion end of the endoscope;
the wedge-shaped lens group 3 is arranged at the far end side of the tube assembly 1, the liquid lens group 2 is arranged at a distance from the wedge-shaped lens 3, and the driver group 4 is electrically connected with the liquid lens group 2;
wherein, the driver 4 group is used for adjusting the curvature of the liquid lens group 2 to realize the focal length adjustment of the endoscope, and the wedge-shaped lens group 3 is used for increasing the imaging angle of the endoscope.
Specifically, the tube assembly described in the embodiments of the present application should be provided with openings on both the proximal and distal sides and through the interior of the tube assembly, and the tubular diameter of the tube assembly should be slightly larger than the insertion end of the endoscope so that the endoscope can be inserted from the proximal side of the tube assembly.
The proximal end of the tube assembly described in the embodiments of the present application is the section that is connected to the endoscope during use, while the corresponding other end of the tube assembly is the distal side.
In an embodiment of the application, the proximal side of the tube assembly is adapted to be detachably connected to the insertion end of the endoscope, i.e. the endoscope can be connected to the accessory via the proximal side of the tube assembly, or can be disconnected from the accessory after connection.
The wedge-shaped lens group described in the embodiments of the present application may be composed of one or more wedge-shaped lenses, and a single wedge-shaped lens may deflect light by θdFor example, the wedge-shaped mirror used in the wedge-shaped mirror group in the embodiment of the present application has a tangential angle of 18.8 ° and a deflection angle of 10 °.
Alternatively, a single wedge mirror may deflect the light by θdAnd two pieces of wedge-shaped mirrors can deflect the light by 4 thetad. For example, the wedge-shaped mirror used in the wedge-shaped mirror group in the embodiment of the present application has a tangential angle of 18.8 ° and a deflection angle of 10 °, i.e. a deflection of 40 ° of light can be achieved.
The wedge-shaped mirror assembly may in particular be arranged on the distal side of the tube assembly, such that the wedge-shaped mirror assembly therein is capable of deflecting light, thereby assisting the endoscope in acquiring a larger field of view.
The set of drives described in the embodiments of the present application may be specifically arranged outside the tube assembly, and may be specifically connected to a computer by means of USB and CMOS wires placed in parallel at the rear, so as to reduce the impact on the volume of the accessory, and there may be one or more drives in the set, each of which may be connected to one or more liquid lenses, and output different driving voltages to the liquid lenses according to the respective specifications, so as to adjust the curvature thereof.
The liquid lens group described in the embodiment of the present application may specifically include one liquid lens or a plurality of liquid lenses, each liquid lens is correspondingly connected to one or more drivers in a driver group, and the curvature of the liquid lens is adjusted by different driving voltages through the driving voltage transmitted by the drivers to the liquid lens, so that the liquid lens group may effectively implement the adjustment of the focal length and the magnification of the endoscope.
In particular, the front end of the outer end side of the tube assembly described in the embodiments of the present application is provided with a lighting device, which may be in particular an LED lamp, or other lighting device, while the lighting device is arranged so as not to obstruct the light acquisition of the wedge-shaped lens group.
In the embodiment of the application, the scope of the surgical field can be effectively enlarged through the wedge-shaped lens group arranged on the front side of the insertion end of the endoscope, and the liquid lens and the corresponding driver group thereof can realize rapid focusing under the condition of freely switching the magnification, thereby ensuring clear and continuous observation of the surgical field region.
Optionally, the liquid lens group includes a first liquid lens and a second liquid lens, and the driver group includes a first driver and a second driver;
the first driver is electrically connected with the first liquid lens, and the second driver is electrically connected with the second liquid lens;
the first liquid lens is used for focal length adjustment, and the second liquid lens is used for magnification adjustment.
Specifically, the liquid lens group described in the embodiments of the present application may include two small-aperture liquid lenses, which are a first liquid lens for focusing and a second liquid lens for method magnification adjustment, respectively.
And correspondingly, the first driver is electrically connected with the first liquid lens, and the second driver is electrically connected with the second liquid lens.
The first driver outputs driving voltage to the first liquid lens, so that the adjustment of the focal length is realized, namely the first driver outputs different driving voltages to correspondingly adjust the curvature of the first liquid lens, and the focal length of the endoscope is continuously adjusted.
The second driver outputs driving voltage to the second liquid lens, so that the magnification is adjusted, namely the second driver outputs different driving voltages to adjust the curvature of the second liquid lens, and the magnification of the endoscope is adjusted continuously.
In the embodiment of the application, the change of the visual angle of an object shot by the endoscope and the synchronous focusing can be effectively realized through the first liquid lens and the second liquid lens.
Optionally, the accessory further comprises: a micro-motor;
the micro motor is connected with the wedge-shaped mirror group and used for controlling the wedge-shaped mirror group to adjust the inclination angle.
Specifically, the micro-motor described in the embodiment of the present application may be electrically connected to each wedge-shaped mirror in the wedge-shaped mirror group, and the angle of each wedge-shaped mirror in the wedge-shaped mirror group is controlled by the micro-motor, and the addition of the wedge-shaped mirror may satisfy the requirementThe incident angle of the light is changed to enlarge the view angle of the operative field. The single wedge-shaped mirror can deflect the light beam by thetadAnd two pieces of wedge-shaped mirrors can deflect the light by 4 thetad。
For example, the wedge-shaped mirror used in the wedge-shaped mirror group in the embodiment of the present application has a tangential angle of 18.8 ° and a deflection angle of 10 °, i.e. a deflection of 40 ° of light can be achieved.
In the embodiment of the application, the micro motor is controlled to control the inclination angle of the middle wedge-shaped mirror of the wedge-shaped mirror group, so that the change of the incident angle of the light of the endoscope can be met, and wide visual fields can be provided for the endoscope.
The wedge-shaped mirror is driven by the motor to rotate, and a plurality of images at different visual angles are shot. Reading the focal length parameter of the liquid lens when each image is shot to obtain the corresponding camera internal reference matrix as follows:
where f is the focal length, dx is the pixel size in the horizontal direction, dy is the pixel size in the vertical direction, (u)o，vo) Is the intersection of the optical axis and the camera sensor.
Feature point detection and matching based on SIFT are carried out on the two images, and the homogeneous coordinates (x) of the corresponding feature points in the two images can be obtained1，y11) and (x)2，y2And 1), obtaining x 'Fx as 0 according to the epipolar geometric relationship, wherein x' and x correspond to the coordinates of the characteristic points in the two images, and obtaining a basic parameter value F of the camera according to an equation.
According to the camera internal parameter matrixes K1 and K2 and the basic parameter value F corresponding to the two images obtained above, the following formula is adopted
E＝K′2*F*K1
The intrinsic matrix E can be obtained and then subjected to singular value decomposition, and E ═ U × Λ × V 'can be obtained, where the third column of U is the translation vector between the two images and U × W × V' is the rotation vector of the images. The two images can be spliced according to the translation and rotation vectors between the two images. And 4 images at different angles are shot in sequence and spliced.
Optionally, the accessory further comprises: a concave lens group and a convex lens group;
the concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group;
the concave lens group is used for eliminating chromatic aberration, and the convex lens group is used for focusing light.
In particular, the attachment described in the embodiments of the present application may have some chromatic aberration due to the use of a variety of different lenses, and therefore the embodiments of the present application also contemplate a concave lens group designed between the proximal side of the tube assembly and the liquid lens group to eliminate chromatic aberration.
The concave lens group may specifically comprise one or more fixed focal length concave lenses, which should be disposed in close proximity to the interface of the endoscope to ensure that the light finally incident into the endoscope has been effectively free of chromatic aberration.
In the embodiment of the present application, in order to control the overall focal length range, a convex lens group may be disposed between the concave lens group and the wedge lens group, and in the case of one or more convex lenses in the convex lens group, the convex lens groups are not necessarily disposed together, for example, in the case of two convex lenses in the convex lens group, the convex lens groups may be disposed respectively in front and rear positions of the liquid lens group.
In the embodiment of the application, chromatic aberration can be effectively eliminated through the concave lens group and the convex lens group, and focal length control is realized, so that better imaging of an endoscope is facilitated.
Optionally, the accessory further comprises: the first input unit and the second input unit are connected with the liquid lens group;
wherein the first input unit is configured to adjust a curvature of the liquid lens group in response to an increase magnification input to magnify an image of the endoscope;
the second input unit is used for responding to the input of the reduction magnification, and adjusting the curvature of the liquid lens group so as to reduce the imaging of the endoscope.
Specifically, the first input unit and the second input unit described in the embodiments of the present application may specifically be two input buttons, and when the user presses the first input unit, it is equivalent to receiving an input of increasing magnification, and when the user presses the second input unit, it is equivalent to receiving an input of decreasing magnification.
For example, in the embodiment of the present application, the focal length and the magnification are set in advance according to the common working distance (100mm) of the laparoscopic minimally invasive surgery. The first piece of the two liquid lenses is used for adjusting the focal length, and the second piece of the liquid lenses is used for adjusting the magnification. The voltage value of the first liquid lens in the first magnification working mode is set to be 24V, the working voltage of the second liquid lens is set to be 66.52V, and the magnification is set to be 1.44. And setting the voltage value of the second liquid lens to be 59.68V, the working voltage of the second liquid lens to be 24V and the magnification to be 0.68 in the second magnification working mode. Two buttons are arranged above the integrated endoscope, so that the two amplification working modes can be rapidly switched in the using process.
In the embodiment of the application, through the first input unit and the second input unit, a user can conveniently and quickly manually adjust the magnification of the endoscope, so that the magnification required by the user can be quickly acquired.
Optionally, a third liquid lens is further included in the liquid lens group, and a third driver is further included in the driver group;
the third driver is electrically connected with the third liquid lens;
the third liquid lens and the first liquid lens are commonly used for focal length adjustment.
As another alternative, the liquid lens group described in the embodiments of the present application may also include three liquid lenses, and may have a larger magnification range and working distance on the basis of 2 liquid lenses.
As an alternative embodiment, the liquid lens group described in the embodiment of the present application can also use only one liquid lens, which can realize the auto-focusing function, and can solve the problem that the traditional endoscope is out of focus when the working distance is less than 30 mm.
Fig. 2 is a schematic structural diagram of an endoscope system provided by an embodiment of the present application, and as shown in fig. 2, the endoscope system includes an endoscope 5 and a tube assembly, and the endoscope 5 is detachably connected to an inner end of the tube assembly.
Fig. 3 is a second structural schematic diagram of the endoscope system described in the embodiment of the present application, as shown in fig. 3, including: the endoscope comprises two small-aperture liquid lenses 31, two convex lenses 33 with fixed focal lengths, two concave lenses 34 with fixed focal lengths and two wedge-shaped lenses 35, wherein the endoscope 5 consists of a CMOS and the convex lenses with fixed focal lengths. The first liquid lens is used for focusing, and the second liquid lens is used for switching the magnification, so that the visual angle change and synchronous focusing of an object shot by the endoscope are realized. The convex lens with the fixed focal length is used for controlling the comprehensive focal length, the concave lens is used for eliminating chromatic aberration generated by the system, the wedge-shaped lens is connected with the micro motor 36 and used for changing the working inclination angle to enlarge the surgical field, the system realizes the magnification of calibrating two working modes to realize the imaging under the enlarging and reducing visual angles, and the quick switching can be carried out through the two buttons 32 on the shell.
Fig. 4 is a schematic flowchart of a control method provided in an embodiment of the present application, and as shown in fig. 4, the control method includes:
the endoscope imaging described in the embodiment of the application refers to an image shot by an endoscope at present, edge detection is performed on an edge area of the endoscope imaging, and the definition of pixel points in the edge area is judged, so that whether the endoscope imaging is fuzzy or not is judged.
Specifically, if a blur occurs during shooting, the picture is affected by a blur radius as shown in the following formula:
where c is the blur radius, f0Is the current focal length value, d is the depth value in the focusing state corresponding to the focal length, dfN is the aperture value for the actual depth value.
And carrying out real-time edge detection on the region of interest in the shot image by using an edge detection Sobel operator. The Sobel operator performs convolution on the x-axis direction and the y-axis direction respectively, and the convolution kernels are as follows:
the partial derivatives in the horizontal and vertical directions are obtained by operation of the two convolution kernels. Finally, the definition degree of each pixel point is measured by the square difference of the two:
because the definition of the image and the focal length are in a Gaussian distribution curve relationship under the same object distance, the extreme value search is carried out on the detected edge through algorithm iteration.
Since the object distance is not changed and the magnification is not adjusted, it is only necessary to determine the target driving voltage of the liquid lens for focusing in the liquid lens group.
And performing search approximation by adopting a golden ratio segmentation method. At each iteration, the algorithm compares the resolution of the 0.38 and 0.62 positions in the interval between the two voltage values. Depending on which of these two points has the best sharpness, the new interval is reduced to the left or right by 0.38 of the previous interval. At each iteration, the span is reduced to a known amount (0.62), so that the convergence criterion can be determined as: regardless of the position of the starting point and the target focus, once the depth of field is fixed, the number of iterations is also fixed. The specific flow of the algorithm is as follows: during the first iteration, calculating the definition of the Sobel operators under the voltage values of 0.618 and 0.38 between the maximum value and the minimum value, taking the larger value between the maximum value and the minimum value as the boundary point of the next round, and in the second iteration and the subsequent iteration, adopting the voltage value of the previous step as the maximum value, repeatedly calculating the definition of the Sobel operators under the voltage values of 0.382 and 0.618 between the maximum value and the minimum value until the definition search range is less than 0.02V, and stopping the search. After n iterations, the error value is:
Err＝(1-R)(n-1)*(Max-Min) (4)
max is the maximum adjustable diopter of the liquid lens, Min is the minimum adjustable diopter of the liquid lens, n is the total iteration number, and R is the error value of each time.
And taking the voltage value of the last iteration as the target driving voltage of the liquid lens group. The liquid lens group forms a continuous focal length change range by calibrating different voltage combination values of two end channels, and the combined focal length between target driving voltages can be obtained by the following formula:
wherein A is the distance between the first liquid lens and the object, B is the distance between the two liquid lenses, C is the distance between the second liquid lens and the imaging sheet,
is the ratio of the image distance and the object distance of the second liquid lens,
is the sum of the image distances of the first liquid lensThe ratio of the object distances.
And because the commonly used magnification data of the endoscope is limited in the use process, the combined voltage corresponding to each magnification can be generated in advance and stored in a table, and in the use process, the respectively corresponding voltage values of the two liquid lenses corresponding to the required combined voltage can be obtained by looking up the table. The whole process usually requires 8-12 pictures to be taken to accomplish intraoperative real-time, high-speed focusing of organs and instruments.
And 430, adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
In particular, after the curvature adjustment of the liquid filter is completed, the endoscope can be focused better at this time, so that an image with higher sharpness is acquired.
In the embodiment of the application, an observer can clearly and continuously observe the operation field and the three-dimensional superposed image in a larger field range under the working mode of freely switching the magnification.
Optionally, the performing sharpness detection on the edge region imaged by the endoscope includes:
under the condition that the endoscope detects that the object distance changes, performing definition detection on an edge area imaged by the endoscope;
or; under the condition that the magnification of the liquid lens group is changed, the definition detection is carried out on the edge area imaged by the endoscope.
Specifically, the object distance described in the embodiment of the present application may be a new object in the visual field of the endoscope, or a large change in the object distance between the object and the lens.
Alternatively, in the case of a significant change in the magnification of the liquid lens group, the imaging of the endoscope may also be blurred, and refocusing may also be required.
In the embodiment of the application, under the condition that the endoscope detects that the object distance changes or the magnification changes, the definition of the edge area imaged by the endoscope is detected again, so that focusing is performed again, and the endoscope can continuously acquire clear images.
Optionally, in this embodiment of the present application, an actual length of an object captured during an endoscope operation may be estimated, and when a curvature of the liquid lens group is adjusted based on the target driving voltage, a clearest point in this state is obtained, and a combined focal length value of the point and the liquid lens group is used as an input, and according to a principle of triangulation of an optical system, a ratio of an object length and a time length in an image may be obtained as an inverse of a pixel density of a camera sensor:
wherein d is3DIs the actual length of the object, d2DIs the length of the object in the image, alpha is the pixel density of the camera sensor, d0F is the focal length parameter of the camera for the depth value in the current focusing mode. From this relationship, the actual length values of the two points cut in the image can be obtained.
In the embodiment of the application, the real information of the target object can be effectively observed and known by the triangulation principle of the optical system.
Optionally, acquiring a plurality of groups of image groups under different wedge mirror angles, which are acquired in the process of searching and approaching the driving voltage of the driver group in a preset voltage interval, wherein each group of image groups comprises at least two images under different magnifications;
SIFT feature point detection and matching are carried out on each picture, and translation vectors and rotation vectors among the images are determined;
and splicing each group of image groups based on the translation vector and the rotation direction to obtain a target image.
The wedge-shaped mirror is driven by the motor to rotate, and a plurality of images at different visual angles are shot. Reading the focal length parameter of the liquid lens when each image is shot to obtain the corresponding camera internal reference matrix as follows:
where f is the focal length, dx is the pixel size in the horizontal direction, dy is the pixel size in the vertical direction, (u)o，vo) Is the intersection of the optical axis and the camera sensor.
Feature point detection and matching based on SIFT are carried out on the two images, and the homogeneous coordinates (x) of the corresponding feature points in the two images can be obtained1，y11) and (x)2，y2And 1), obtaining x 'Fx as 0 according to the epipolar geometric relationship, wherein x' and x correspond to the coordinates of the characteristic points in the two images, and obtaining a basic parameter value F of the camera according to an equation.
According to the camera internal parameter matrixes K1 and K2 and the basic parameter value F corresponding to the two images obtained above, the following formula is adopted
E＝K′2*F*K1
The intrinsic matrix E can be obtained and then subjected to singular value decomposition, and E ═ U × Λ × V 'can be obtained, where the third column of U is the translation vector between the two images and U × W × V' is the rotation vector of the images. The two images can be spliced according to the translation and rotation vectors between the two images. And meanwhile, according to the definition of the projection matrix, the final three-dimensional coordinates of the feature matching point pairs can be calculated. And sequentially calculating the three-dimensional coordinates of each image according to the steps, and splicing the three-dimensional point clouds to obtain the corresponding large-view three-dimensional image.
Optionally, after the step of determining a translation vector and a rotation vector between the images, further comprising;
constructing a first depth map based on the translation vector and the rotation vector;
performing focusing edge detection on each image by using a Laplace Gaussian kernel to obtain a second depth map, wherein the corresponding depth value of each pixel in the second depth map is determined based on a layer where the norm maximum value of the Laplace Gaussian kernel is located;
normalizing the second depth map, and performing depth optimization processing on the second depth map to obtain a third depth map;
adding the first depth map and the third depth map, and averaging to obtain a fourth fused depth map;
splicing the fourth fusion depth map based on the translation vector and the rotation vector to obtain a target fusion map;
and determining a target three-dimensional reconstruction model based on the target fusion depth map and three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on the adjusted imaging of the liquid lens group.
When automatic focusing is carried out, a plurality of images collected in the focusing process are images of the same object under different focal lengths, the out-of-focus degree of the sequence of images is different, and the algorithm implementation mode in the depth estimation and three-dimensional reconstruction module is as follows: according to the known focal length values, the camera internal reference matrixes corresponding to the two images can be obtained as follows:
where f is the focal length, dx is the pixel size in the horizontal direction, dy is the pixel size in the vertical direction, (u)o，vo) Is the intersection of the optical axis and the camera sensor.
Feature point detection and matching based on SIFT are carried out on the two images, and the homogeneous coordinates (x) of the corresponding feature points in the two images can be obtained1，y11) and (x)2，y2And 1), obtaining x 'Fx as 0 according to the epipolar geometric relationship, wherein x' and x correspond to the coordinates of the characteristic points in the two images, and obtaining a basic parameter value F of the camera according to an equation.
According to the camera internal parameter matrixes K1 and K2 and the basic parameter value F corresponding to the two images obtained above, the following formula is adopted
E＝K′2*F*K1
The intrinsic matrix E can be obtained and then subjected to singular value decomposition, and E ═ U × Λ × V 'can be obtained, where the third column of U is the translation vector between the two images and U × W × V' is the rotation vector of the images. According to the translation and rotation vectors between the two images, the two images can be registered, and meanwhile, according to the definition of a projection matrix, the final three-dimensional coordinates of the feature matching point pairs and the corresponding first depth map can be calculated.
Performing focus edge detection on the image by Laplace Gaussian matching to calculate focus degree fmThe implementation formula of (2) is as follows:
wherein LOG is laplacian gaussian kernel, i (x, y) represents the contrast of the image, N represents the window size, and (i, j) represents the currently calculated pixel. And calculating the norm of the pixel according to the calculation result, wherein the layer where the maximum norm value of each pixel (i, j) is located is the depth layer where the point is located. And obtaining a depth map corresponding to the whole image through normalization, and performing depth optimization operations such as hole filling, mean value filtering and the like on the depth map to obtain a second depth map.
Adding, averaging and processing the first depth map and the second depth map to obtain a target fusion depth map; splicing the fusion depth map according to rotation and translation matrixes among the images under different angles of the wedge-shaped mirror to obtain a fusion depth map with a larger visual angle;
after the automatic focusing module adjusts to the clearest point, the image shot under the voltage value is a full-focus image, and the image is used as texture input of depth estimation and three-dimensional reconstruction. And performing three-dimensional reconstruction according to the obtained depth map, and covering the obtained full-focus image with textures to obtain a three-dimensional reconstruction model of the target object.
Fig. 5 is a schematic structural diagram of a control device according to an embodiment of the present application, and as shown in fig. 5, the control device includes:
the definition measuring module 510 is used for performing definition detection on the edge region imaged by the endoscope;
the extreme value searching module 520 is configured to obtain a target driving voltage of the driver set by comparing the contrast strength of the current image and the previous image and performing search approximation on the driving voltage of the driver set in a preset voltage interval by using a golden ratio segmentation method, where the target driving voltage is a driving voltage corresponding to the highest definition of the pixel points in the edge region;
the adjusting module 530 is configured to adjust the curvature of the liquid lens group based on the target driving voltage, so as to form an image according to the adjusted liquid lens group.
In the embodiment of the application, the scope of the surgical field can be effectively enlarged through the wedge-shaped lens group arranged on the front side of the insertion end of the endoscope, and the liquid lens and the corresponding driver group thereof can realize rapid focusing under the condition of freely switching the magnification, thereby ensuring clear and continuous observation of the surgical field region.
Fig. 6 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 6: a processor (processor)610, a communication Interface (Communications Interface)620, a memory (memory)630 and a communication bus 640, wherein the processor 610, the communication Interface 620 and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call logic instructions in the memory 630 to perform a control method comprising: performing definition detection on an edge region imaged by an endoscope; obtaining a target driving voltage of the driver set by comparing the contrast of the current image and the previous frame of image and searching and approaching the driving voltage of the driver set in a preset voltage interval by using a golden ratio segmentation method, wherein the target driving voltage is the driving voltage corresponding to the highest definition of the pixel points in the edge region; and adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
In addition, the logic instructions in the memory 630 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer-readable storage medium, the computer program, when executed by a processor, being capable of executing the control method provided by the above methods, the method comprising: under the condition that the definition of the pixel points in the edge area is lower than a preset threshold, searching and approaching the driving voltage of the driver set in a preset voltage interval by a golden ratio division method to obtain a target driving voltage of the driver set, wherein the target driving voltage is the driving voltage corresponding to the pixel points in the edge area when the definition is the highest; and adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a control method provided by performing the above methods, the method including: under the condition that the definition of the pixel points in the edge area is lower than a preset threshold, searching and approaching the driving voltage of the driver set in a preset voltage interval by a golden ratio division method to obtain a target driving voltage of the driver set, wherein the target driving voltage is the driving voltage corresponding to the pixel points in the edge area when the definition is the highest; and adjusting the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (11)
1. An accessory for an endoscope, the accessory comprising: the device comprises a tube assembly, a liquid lens group, a wedge-shaped lens group and a driver group;
the proximal side of the tube assembly is for detachable connection with an insertion end of the endoscope;
the wedge-shaped lens group is arranged on the far end side of the tube assembly, the liquid lens group and the wedge-shaped lens are arranged at intervals, and the driver group is electrically connected with the liquid lens group;
the driver group is used for adjusting the curvature of the liquid lens group so as to adjust the focal length and the magnification of the endoscope, and the wedge-shaped lens group is used for increasing the imaging angle of the endoscope.
2. The attachment for an endoscope of claim 1, wherein the liquid lens group includes a first liquid lens and a second liquid lens, the driver group includes a first driver and a second driver;
the first driver is electrically connected with the first liquid lens, and the second driver is electrically connected with the second liquid lens;
the first liquid lens is used for focal length adjustment, and the second liquid lens is used for magnification adjustment.
3. An accessory for an endoscope according to claim 1 and also comprising: a micro-motor;
the micro motor is connected with the wedge-shaped mirror group and used for controlling the wedge-shaped mirror group to adjust the inclination angle.
4. An accessory for an endoscope according to claim 1 and also comprising: a concave lens group and a convex lens group;
the concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group;
the concave lens group is used for eliminating chromatic aberration, and the convex lens group is used for focusing light.
5. An accessory for an endoscope according to claim 1 and also comprising: the first input unit and the second input unit are connected with the liquid lens group;
wherein the first input unit is configured to adjust a curvature of the liquid lens group in response to an increase magnification input to magnify an image of the endoscope;
the second input unit is used for responding to the input of the reduction magnification, and adjusting the curvature of the liquid lens group so as to reduce the imaging of the endoscope.
6. An attachment for an endoscope according to claim 2 and wherein said liquid lens group further comprises a third liquid lens, and said actuator group further comprises a third actuator;
the third driver is electrically connected with the third liquid lens;
the third liquid lens and the first liquid lens are commonly used for focal length adjustment.
7. An endoscopic system comprising an endoscope and an accessory for an endoscope according to any of claims 1 to 6.
8. A control method of an endoscope system according to claim 7, comprising:
performing definition detection on an edge region imaged by an endoscope;
under the condition that the definition of the pixel points in the edge area is lower than a preset threshold, searching and approaching the driving voltage of the driver set in a preset voltage interval by a golden ratio division method to obtain a target driving voltage of the driver set, wherein the target driving voltage is the driving voltage corresponding to the pixel points in the edge area when the definition is the highest;
and adjusting the curvature of the liquid lens group based on the target driving voltage to obtain the image of the adjusted liquid lens group.
9. The control method according to claim 8, wherein the performing sharpness detection on the edge region imaged by the endoscope comprises:
under the condition that the endoscope detects that the object distance changes, performing definition detection on an edge area imaged by the endoscope;
or; under the condition that the magnification of the liquid lens group is changed, the definition detection is carried out on the edge area imaged by the endoscope.
10. The control method according to claim 8, wherein, at the adjusting of the curvature of the liquid lens group based on the target driving voltage to image according to the adjusted liquid lens group, the method further comprises:
acquiring a plurality of groups of image groups under different wedge-shaped mirror angles, which are acquired in the process of searching and approaching the driving voltage of the driver group in a preset voltage interval, wherein each group of image groups comprises at least two images under different magnifications;
SIFT feature point detection and matching are carried out on each image, and translation vectors and rotation vectors among the images are determined;
and splicing each group of image groups based on the translation vector and the rotation direction to obtain a target image.
11. The control method according to claim 10, further comprising, after the step of determining a translation vector and a rotation vector between the respective images;
constructing a first depth map based on the translation vector and the rotation vector;
performing focusing edge detection on each image by using a Laplace Gaussian kernel to obtain a second depth map, wherein the corresponding depth value of each pixel in the second depth map is determined based on a layer where the norm maximum value of the Laplace Gaussian kernel is located;
normalizing the second depth map, and performing depth optimization processing on the second depth map to obtain a third depth map;
adding the first depth map and the third depth map, and averaging to obtain a fourth fused depth map;
splicing the fourth fusion depth map based on the translation vector and the rotation vector to obtain a target fusion map;
and determining a target three-dimensional reconstruction model based on the target fusion depth map and three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on the adjusted imaging of the liquid lens group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111501595.5A CN114081420A (en) | 2021-12-09 | 2021-12-09 | Accessory for endoscope, endoscope system, control method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111501595.5A CN114081420A (en) | 2021-12-09 | 2021-12-09 | Accessory for endoscope, endoscope system, control method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114081420A true CN114081420A (en) | 2022-02-25 |
Family
ID=80307227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111501595.5A Pending CN114081420A (en) | 2021-12-09 | 2021-12-09 | Accessory for endoscope, endoscope system, control method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114081420A (en) |
-
2021
- 2021-12-09 CN CN202111501595.5A patent/CN114081420A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5284731B2 (en) | Stereoscopic image display system | |
| US10327627B2 (en) | Use of plenoptic otoscope data for aiding medical diagnosis | |
| US10992917B2 (en) | Image processing device, image processing method, program, and image processing system that use parallax information | |
| EP3142540B1 (en) | Colposcopy apparatus for performing a colposcopy procedure | |
| JP6828695B2 (en) | Medical image processing equipment, systems, methods and programs | |
| CN104939793A (en) | Variable-focus 3-D capsule endoscope system based on liquid lens | |
| US11974053B2 (en) | Stereoscopic imaging platform with continuous autofocusing mode | |
| US20240073399A1 (en) | Stereoscopic imaging platform with disparity and sharpness control automatic focusing mode | |
| CN114081420A (en) | Accessory for endoscope, endoscope system, control method and device | |
| WO2014208224A1 (en) | Autofocus device | |
| US11863882B2 (en) | Stereoscopic imaging platform with target locking automatic focusing mode | |
| US20240236515A1 (en) | Stereoscopic imaging platform with continuous autofocusing mode | |
| JP2020162926A (en) | Ophthalmology imaging apparatus and ophthalmology image processing program | |
| EP3119264B1 (en) | Optically adaptive endoscope | |
| EP4000494B1 (en) | Imaging system, laparoscope and method for imaging an object | |
| US20240069320A1 (en) | Digital microscope for medical procedure | |
| CN114859544A (en) | Endoscope focusing control method and system | |
| CN118340496A (en) | Automatic positioning and collecting device and method for clinic instrument, electronic equipment and storage medium | |
| CN117768640A (en) | Image synthesis detection system and method for segmented imaging and multiple depth of field |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |