US20120224026A1

US20120224026A1 - System and method for producing and improving images

Info

Publication number: US20120224026A1
Application number: US13/467,909
Authority: US
Inventors: Lex Bayer; Michael Stewart
Original assignee: Avantis Medical Systems Inc
Current assignee: Psip LLC
Priority date: 2006-05-19
Filing date: 2012-05-09
Publication date: 2012-09-06
Also published as: WO2007136879A3; US20130197304A1; WO2007136859A2; WO2007136879B1; US20070279486A1; JP2009537284A; US8587645B2; US20140046136A1; JP2009537283A; WO2007136859A3; WO2007136879A2; EP2023794A2; US20070270642A1; US8197399B2; EP2023795A2; US8310530B2

Abstract

A method for displaying images includes adjusting at least one characteristic of an image from a first imaging device of an endoscope to match at least one corresponding characteristic of an image from a second imaging device of the endoscope. The at least one characteristic may be one or more of color, contrast and brightness. An endoscopic system includes an endoscope including a first imaging device and a second imaging device, and a display device that displays an image from the first imaging device of the endoscope and an image from the second imaging device of the endoscope, wherein the images are sized so that an object, when placed at the same distance from the imaging devices, appears to have about the same size in the images.

Description

This application claims the benefit of U.S. Provisional Patent Application No. 60/801,748, filed May 19, 2006, the entire disclosure of which is incorporated herein by reference.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to a system and method for producing and improving images.

BACKGROUND OF THE INVENTION

Multiple endoscopic devices with multiple cameras and light sources may be used for medical procedures, inspection of small pipes, or remote monitoring. For example, such an endoscopic device may be a medical endoscope comprising a flexible tube, and a camera and a light source mounted on the distal end of the flexible tube. The endoscope is insertable into an internal body cavity through a body orifice to examine the body cavity and tissues for diagnosis. The tube of the endoscope has one or more longitudinal channels, through which an instrument can reach the body cavity to take samples of suspicious tissues or to perform other surgical procedures such as polypectomy.
There are many types of endoscopes, and they are named in relation to the organs or areas with which they are used. For example, gastroscopes are used for examination and treatment of the esophagus, stomach and duodenum; colonoscopes for the colon; bronchoscopes for the bronchi; laparoscopes for the peritoneal cavity; sigmoidoscopes for the rectum and the sigmoid colon; arthroscopes for joints; cystoscopes for the urinary bladder; and angioscopes for the examination of blood vessels.
Each endoscope has a single forward viewing camera mounted at the distal end of the flexible tube to transmit an image to an eyepiece or video camera at the proximal end. The camera is used to assist a medical professional in advancing the endoscope into a body cavity and looking for abnormalities. The camera provides the medical professional with a two-dimensional view from the distal end of the endoscope. To capture an image from a different angle or in a different portion, the endoscope must be repositioned or moved back and forth. Repositioning and movement of the endoscope prolongs the procedure and causes added discomfort, complications, and risks to the patient. Additionally, in an environment similar to the lower gastro-intestinal tract, flexures, tissue folds and unusual geometries of the organ may prevent the endoscope's camera from viewing all areas of the organ. The unseen area may cause a potentially malignant (cancerous) polyp to be missed.
This problem can be overcome by providing an auxiliary camera and an auxiliary light source. The auxiliary camera and light source can be oriented to face the main camera and light source, thus providing an image of areas not viewable by the endoscope's main camera. This arrangement of cameras and light sources can provide both front and rear views of an area or an abnormality. In the case of polypectomy where a polyp is excised by placing a wire loop around the base of the polyp, the camera arrangement allows better placement of the wire loop to minimize damage to the adjacent healthy tissue.

SUMMARY OF THE INVENTION

The present invention relates to devices and methods for producing and improving video images generated by the imaging devices of endoscopes.
In accordance with one aspect of the invention, a method for displaying images includes adjusting at least one characteristic of an image from a first imaging device of an endoscope to match at least one corresponding characteristic of an image from a second imaging device of the endoscope. The characteristic may be one or more of color, contrast and brightness.
In a preferred embodiment, the adjusting step includes creating a histogram for each of RGB colors for the image from the first imaging device and a histogram for each of the RGB colors for the image from the second imaging device; adjusting the gamut of each histogram of the image from the first imaging device to match the gamut of the corresponding histogram of the image from the second imaging device; and using gamma coefficients to adjust a color level of each histogram of the image from the first imaging device to match a color level of the corresponding histogram of the image from the second imaging device.
In accordance with another aspect of the invention, a method for displaying images includes placing, side by side, an image from a first imaging device of an endoscope and an image from a second imaging device of the endoscope, wherein the imaging devices face each other; and reversing one of the images left for right.
In accordance with still another aspect of the invention, a method for sizing images includes placing an image from a first imaging device of an endoscope and an image from a second imaging device of the endoscope on a display device; and sizing the images so that an object, when placed at the same distance from the imaging devices, appears to have about the same size in the images.
In accordance with yet another aspect of the invention, a method for processing images includes placing image data from first and second imaging devices of an endoscope in one computer file for simultaneous display on a display device. Preferably, the image data from the imaging devices are time-correlated.
In a preferred embodiment, patient information data is also placed in the computer file for simultaneous display with the images on the display device.
In a further preferred embodiment, a time stamp is placed in the computer file for simultaneous display with the images and patient information data on the display device.
In accordance with still yet another aspect of the invention, an endoscopic system includes an endoscope that has a first imaging device and a second imaging device, and a controller that adjusts at least one characteristic of an image from the first imaging device of the endoscope to match at least one corresponding characteristic of an image from the second imaging device of the endoscope. The at least one characteristic may be one or more of color, contrast and brightness.
In a preferred embodiment, the controller creates a histogram for each of RGB colors for the image from the first imaging device and a histogram for each of the RGB colors for the image from the second imaging device; adjusts the gamut of each histogram of the image from the first imaging device to match the gamut of the corresponding histogram of the image from the second imaging device; and uses gamma coefficients to adjust a color level of each histogram of the image from the first imaging device to match a color level of the corresponding histogram of the image from the second imaging device.
In accordance with a further aspect of the invention, an endoscopic system includes an endoscope including a first imaging device and a second imaging device, and a display device that displays, side by side, an image from the first imaging device of the endoscope and an image from the second imaging device of the endoscope, wherein the imaging devices face each other, and wherein one of the images is reversed left for right.
In accordance with a still further aspect of the invention, an endoscopic system includes an endoscope including a first imaging device and a second imaging device, and a display device that displays an image from the first imaging device of the endoscope and an image from the second imaging device of the endoscope, wherein the images are sized so that an object, when placed at the same distance from the imaging devices, appears to have about the same size in the images.
In accordance with a yet further aspect of the invention, an endoscopic system includes an endoscope including a first imaging device and a second imaging device, and a controller that places image data from the first and second imaging devices of the endoscope in one computer file for simultaneous display on a display device. Preferably, the image data from the imaging devices are time-correlated.
In a preferred embodiment, patient information data is also placed in the computer file for simultaneous display with the images on the display device.
In a further preferred embodiment, a time stamp is placed in the computer file for simultaneous display with the images and patient information data on the display device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a perspective view of an endoscope with an imaging assembly according to one embodiment of the present invention.

FIG. 2 shows a perspective view of the distal end of an insertion tube of the endoscope of FIG. 1.

FIG. 3 shows a perspective view of the imaging assembly shown in FIG. 1.

FIG. 4 shows a perspective view of the distal ends of the endoscope and imaging assembly of FIG. 1.

FIG. 5 shows a schematic representation of a display device used with the endoscope of FIG. 1.

FIG. 6 shows a schematic representation of a screen showing two images and patient information.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

FIG. 1 illustrates an exemplary endoscope 10 of the present invention. This endoscope 10 can be used in a variety of medical procedures in which imaging of a body tissue, organ, cavity or lumen is required. The types of procedures include, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, cystoscopy, EGD, laparoscopy, and sigmoidoscopy.
The endoscope 10 of FIG. 1 includes an insertion tube 12 and an imaging assembly 14, a section of which is housed inside the insertion tube 12. As shown in FIG. 2, the insertion tube 12 has two longitudinal channels 16. In general, however, the insertion tube 12 may have any number of longitudinal channels. An instrument can reach the body cavity through one of the channels 16 to perform any desired procedures, such as to take samples of suspicious tissues or to perform other surgical procedures such as polypectomy. The instruments may be, for example, a retractable needle for drug injection, hydraulically actuated scissors, clamps, grasping tools, electrocoagulation systems, ultrasound transducers, electrical sensors, heating elements, laser mechanisms and other ablation means. In some embodiments, one of the channels can be used to supply a washing liquid such as water for washing. Another or the same channel may be used to supply a gas, such as CO₂or air into the organ. The channels 16 may also be used to extract fluids or inject fluids, such as a drug in a liquid carrier, into the body. Various biopsy, drug delivery, and other diagnostic and therapeutic devices may also be inserted via the channels 16 to perform specific functions.
The insertion tube 12 preferably is steerable or has a steerable distal end region 18 as shown in FIG. 1. The length of the distal end region 18 may be any suitable fraction of the length of the insertion tube 12, such as one half, one third, one fourth, one sixth, one tenth, or one twentieth. The insertion tube 12 may have control cables (not shown) for the manipulation of the insertion tube 12. Preferably, the control cables are symmetrically positioned within the insertion tube 12 and extend along the length of the insertion tube 12. The control cables may be anchored at or near the distal end 36 of the insertion tube 12. Each of the control cables may be a Bowden cable, which includes a wire contained in a flexible overlying hollow tube. The wires of the Bowden cables are attached to controls 20 in the handle 22. Using the controls 20, the wires can be pulled to bend the distal end region 18 of the insertion tube 12 in a given direction. The Bowden cables can be used to articulate the distal end region 18 of the insertion tube 12 in different directions.
As shown in FIG. 1, the endoscope 10 may also include a control handle 22 connected to the proximal end 24 of the insertion tube 12. Preferably, the control handle 22 has one or more ports and/or valves (not shown) for controlling access to the channels 16 of the insertion tube 12. The ports and/or valves can be air or water valves, suction valves, instrumentation ports, and suction/instrumentation ports. As shown in FIG. 1, the control handle 22 may additionally include buttons 26 for taking pictures with an imaging device on the insertion tube 12, the imaging assembly 14, or both. The proximal end 28 of the control handle 22 may include an accessory outlet 30 (FIG. 1) that provides fluid communication between the air, water and suction channels and the pumps and related accessories. The same outlet 30 or a different outlet can be used for electrical lines to light and imaging components at the distal end of the endoscope 10.
As shown in FIG. 2, the endoscope 10 may further include an imaging device 32 and light sources 34, both of which are disposed at the distal end 36 of the insertion tube 12. The imaging device 32 may include, for example, a lens, single chip sensor, multiple chip sensor or fiber optic implemented devices. The imaging device 32, in electrical communication with a processor and/or monitor, may provide still images or recorded or live video images. The light sources 34 preferably are equidistant from the imaging device 32 to provide even illumination. The intensity of each light source 34 can be adjusted to achieve optimum imaging. The circuits for the imaging device 32 and light sources 34 may be incorporated into a printed circuit board (PCB).
As shown in FIGS. 3 and 4, the imaging assembly 14 may include a tubular body 38, a handle 42 connected to the proximal end 40 of the tubular body 38, an auxiliary imaging device 44, a link 46 that provides physical and/or electrical connection between the auxiliary imaging device 44 to the distal end 48 of the tubular body 38, and an auxiliary light source 50 (FIG. 4). The auxiliary light source 50 may be an LED device.
As shown in FIG. 4, the imaging assembly 14 of the endoscope 10 is used to provide an auxiliary imaging device at the distal end of the insertion tube 12. To this end, the imaging assembly 14 is placed inside one of the channels 16 of the endoscope's insertion tube 12 with its auxiliary imaging device 44 disposed beyond the distal end 36 of the insertion tube 12. This can be accomplished by first inserting the distal end of the imaging assembly 14 into the insertion tube's channel 16 from the endoscope's handle 18 and then pushing the imaging assembly 14 further into the assembly 14 until the auxiliary imaging device 44 and link 46 of the imaging assembly 14 are positioned outside the distal end 36 of the insertion tube 12 as shown in FIG. 4.
Each of the main and auxiliary imaging devices 32, 44 may be an electronic device which converts light incident on photosensitive semiconductor elements into electrical signals. The imaging sensor may detect either color or black-and-white images. The signals from the imaging sensor can be digitized and used to reproduce an image that is incident on the imaging sensor. Two commonly used types of image sensors are Charge Coupled Devices (CCD) such as a VCC-5774 produced by Sanyo of Osaka, Japan and Complementary Metal Oxide Semiconductor (CMOS) camera chips such as an OVT 6910 produced by OmniVision of Sunnyvale, Calif. Preferably, the main imaging device 32 is a CCD imaging device, and the auxiliary imaging device 44 is a CMOS imaging device.
When the imaging assembly 14 is properly installed in the insertion tube 12, the auxiliary imaging device 44 of the imaging assembly 14 preferably faces backwards towards the main imaging device 32 as illustrated in FIG. 4. The auxiliary imaging device 44 may be oriented so that the auxiliary imaging device 44 and the main imaging device 32 have adjacent or overlapping viewing areas. Alternatively, the auxiliary imaging device 44 may be oriented so that the auxiliary imaging device 44 and the main imaging device 32 simultaneously provide different views of the same area. Preferably, the auxiliary imaging device 44 provides a retrograde view of the area, while the main imaging device 32 provides a front view of the area. However, the auxiliary imaging device 44 could be oriented in other directions to provide other views, including views that are substantially parallel to the axis of the main imaging device 32.
As shown in FIG. 4, the link 46 connects the auxiliary imaging device 44 to the distal end 48 of the tubular body 38. Preferably, the link 46 is a flexible link that is at least partially made from a flexible shape memory material that substantially tends to return to its original shape after deformation. Shape memory materials are well known and include shape memory alloys and shape memory polymers. A suitable flexible shape memory material is a shape memory alloy such as nitinol. The flexible link 46 is straightened to allow the distal end of the imaging assembly 14 to be inserted into the proximal end of assembly 14 of the insertion tube 12 and then pushed towards the distal end 36 of the insertion tube 12. When the auxiliary imaging device 44 and flexible link 46 are pushed sufficiently out of the distal end 36 of the insertion tube 12, the flexible link 46 resumes its natural bent configuration as shown in FIG. 3. The natural configuration of the flexible link 46 is the configuration of the flexible link 46 when the flexible link 46 is not subject to any force or stress. When the flexible link 46 resumes its natural bent configuration, the auxiliary imaging device 44 faces substantially back towards the distal end 36 of the insertion tube 12 as shown in FIG. 5.
In the illustrated embodiment, the auxiliary light source 50 of the imaging assembly 14 is placed on the flexible link 46, in particular on the curved concave portion of the flexible link 46. The auxiliary light source 50 provides illumination for the auxiliary imaging device 44 and may face substantially the same direction as the auxiliary imaging device 44 as shown in FIG. 4.
The endoscope of the present invention, such as the endoscope 10 shown in FIG. 1, may be part of an endoscope system that may also include a controller 52 and a display device 54, as shown in FIG. 5. In the preferred embodiment shown in FIG. 5, the controller 52 is connected to the main and auxiliary imaging devices 32, 44 to receive image data. The controller 52 may be used to process the image data and transmit the processed image data to the display device 54. The term “controller” as used in this specification is broadly defined. In some embodiments, for example, the controller may simply be a signal processing unit.
In the embodiment shown in FIG. 5, the display device 54 displays, side by side, the image 56 from the main imaging device 32 and the image 58 from the auxiliary imaging device 44. In the present invention, the images may also be displayed on different display devices, and the term “side by side” may simply mean that the two images are positioned so that they can be viewed by the same operator during a medical procedure. The controller 52 preferably incorporates the image data from the main and auxiliary imaging devices 32, 44 into a single signal and sends the signal to the display device 54. In some embodiments, the display device 54 includes a wide screen display with a 16:9 aspect ratio. Preferably, the two images 56, 58 are sized appropriately for display on the wide screen display. For example, the image 56 from the main imaging device 32 may be displayed about 1.5 times larger than the image 58 from the auxiliary imaging device 44. This sizing ratio may also be used to balance the resolution of the two images, as well as to take into account the different aspect ratios of the two images 56, 58. For example, the image 56 from the main imaging device 32 may be displayed with a 1:1 aspect ratio, while the image 58 from the auxiliary imaging device 44 may have a 4:3 aspect ration. The images 56, 58 may also be sized so that the same object, when placed at the same distance from the imaging devices 32, 44, appears to have about the same size in the images 56, 58. The images 56, 58 shown in FIG. 5 are not drawn to scale.
As shown in FIG. 5, one of the images 56, 58 on the display device 54 may be reversed from left for right. With this arrangement, an object 60 that appears on the left side of one image 56 also appears on the left side of the other image 58. Similarly, an object 62 that appears on the right side of one image 56 also appears on the right side of the other image 58, 56. Additionally, when an object moves from the left side of one of the images 56, 58 to the right side, the same object also moves from the left side of the other image 56, 58 to the right side. And, if the object in one image appears to rotated clockwise, the object will appear to rotate clockwise in the other image. Furthermore, the movements of the imaging devices 32, 44 appear to be coordinated. This arrangement makes it easier for an operator to observe, identify and correlate the objects and their movements in both images 56, 58.
Preferably, the data for the two images 56, 58 and possibly other data 64, such as patient information data or a time stamp, are stored in one computer file. In some cases, the patent information may be associated with one of the two images 56, 58. Preferably, the stored images 56, 58 and possibly other data 64 are time-correlated (i.e., they are captured at the same time). For example, as shown in FIG. 6, the two images 56, 58 and possibly other data 64 may be incorporated into one screen 66 in an image file. In some embodiments, the two images 56, 58 and possibly other data 64 may be captured in one jpeg file.
In some preferred embodiments, one or more characteristics of one image 56, 58 may be adjusted to match the same or similar one or more characteristics of the other image 58, 56, so that the images 56, 58 and the objects in the images 56, 58 have similar appearances. The characteristics may include, for example, color, contrast, and brightness. In one example, one or more characteristics of the auxiliary imaging device's image 58 are adjusted to match those of the main imaging device's image 56. Matched images make it easier for an operator to observe, identify and correlate the objects in the images.
In one preferred embodiment, the following technique is used to adjust the characteristics of the auxiliary imaging device's image 58 to match those of the main imaging device's image 56. First, a histogram for each of the RGB colors is created for the auxiliary imaging device's image 58 (called “current file”). The image used to create the histograms may be an average of the past images, such as the past two to ten images, preferably the past four images. And a histogram for each of the RGB colors is created also for the main imaging device's image 56 (called “master file”). This histogram may be the average of the histograms of the past images, such as the histograms of the past two to ten images, preferably the histograms of the past four images.
Second, a minimum and maximum is determined for each histogram by means of thresholding. Then a clip and gain is set for each histogram of the auxiliary imaging device's image to equalize its color gamut to that of the corresponding histogram of the main imaging device's image. In particular, the minimum and maximum for each histogram of the auxiliary imaging device's image are adjusted to match those for the corresponding histogram of the main imaging device's image.
Finally, gamma coefficients are used to adjust the color levels of the histograms of the auxiliary imaging device's image to match those of the histograms of the main imaging device's image. The equations for the gamma coefficients are:

- red_gamma_color_balance=(current_profile.m_AverageRed*master average)/(master_profile.m_AverageRed * current_average);
- green_gamma_color_balance=(current_profile.m_AverageGreen*master_average)/(master_profile.m_AverageGreen * current_average); and
- blue_gamma_color_balance =(current_profile.m_AverageBlue*master_average)/(master_profile.m_AverageBlue * current_average)
  Gamma coefficients are used because they are simple and convenient and preserve black and white points and because the code can be re-used for conventional gamma correction.

Additional processing of the images, such as sharpening, frame averaging, and noise reduction, may be performed.
The images described above may be still pictures or continuous video images (such as television images). When the images are video images, in the embodiment of the invention in which one or more characteristics of one image are adjusted to match those of another image, the characteristics are adjusted continuously in real time (i.e., dynamically). For example, the characteristics of the video image may be adjusted for every frame of the image. The reason for real time adjustment is that the video images are changing constantly as the lighting, object distance or tissue color varies.
The implementation of the above-described features may be performed digitally by software or firmware in the controller. Alternately, the image manipulation can be performed by hardware image chipsets, FPGAs or other electrical circuitry. These image manipulation techniques are well known in the field of graphic and video processing and will not be described in detail.
Although in the preferred embodiments described above, the images are from the main and auxiliary imaging devices of the same endoscope, the images may also come from imaging devices of different endoscopes such as laparoscopes. For example, when two laparoscopes are used during a procedure, the images from the laparoscopes may have different characteristics due to, for example, different imaging device types, different manufacturing techniques, or differences in lighting sensitivities. The controller that receives images from the laparoscopes may designate any one of the images as a master and then match the second image to the master image. In this way the operator is able to conduct a procedure with consistent visualization across the laparoscopes. Additionally, the present invention may be used with three or more images from two or more endoscopes. For example, two images may be adjusted to match a third image.
While particular embodiments of the present invention have been shown and described, it will be obvious to those skilled in the art that changes and modifications can be made without departing from this invention in its broader aspects. Therefore, the appended claims are to encompass within their scope all such changes and modifications as fall within the true spirit and scope of this invention.

Claims

1-36. (canceled)

37. An endoscopic system comprising:

a first imaging sensor of an endoscope;

a second imaging sensor, wherein the second imaging sensor faces the first imaging sensor;

a controller programmed with an algorithm to reverse left for right, an image from either the first imaging sensor or an image from the second imaging sensor; and

a display device that displays, side by side, the images from the first imaging sensor and the second imaging sensor, wherein one of the images is reversed left for right.

38. The endoscopic system of claim 37, wherein the controller reverses the image from either the first imaging sensor or the second imaging sensor such the location of an object being simultaneously viewed by the first and second imaging sensors is correlated in both images.

39. The endoscopic system of claim 38, wherein the object is in the same general location in both images.

40. The endoscopic system of claim 38, wherein the movement of the object is correlated in both images.

41. The endoscopic system of claim 37, wherein the controller is programmed to reverse the image from the first imaging sensor.

42. The endoscopic system of claim 37, wherein the controller is programmed to reverse the image from the second imaging sensor.

43. A method for displaying images, comprising:

acquiring a first image from a first imaging sensor of an endoscope;

acquiring a second image from a second imaging sensor, wherein the second imaging sensor faces the first imaging sensor;

placing, side by side, the first image and the second image on a display device; and

using a controller that has been pre-programmed with an algorithm to reverse one of the images left for right.

44. An endoscopic system comprising:

a first imaging sensor;

a second imaging sensor, wherein the second imaging sensor faces that first imaging sensor;

a controller programmed with an algorithm to size an image from the first imaging sensor and an image from the second imaging sensor so that an object located at the same distance from both the imaging sensors appears to have about the same size in both the images; and

a display device that displays the images from the first imaging sensor and the second imaging sensor, wherein the images are sized by the controller.

45. A method for sizing images, comprising:

acquiring a first image from a first imaging sensor of an endoscope;

placing the first image and the second image on a display device; and

using a controller that has been pre-programmed with an algorithm to size the images so that an object located at the same distance from both the imaging sensors appears to have the same size in both images.

46. An endoscopic system comprising:

a first imaging sensor of an endoscope;

a second imaging sensor, wherein the second imaging sensor faces the first imaging sensor; and

a controller that stores image data from the first and second imaging sensors in one computer file for simultaneous display on a display device.

47. The system of claim 46, wherein the image data from the imaging sensors are time-correlated.

48. The system of claim 46, wherein the controller places patient information data from only one image in the computer file for simultaneous display with the images on a display device.

49. A method for processing images, comprising:

acquiring a first image from a first imaging sensor of an endoscope;

acquiring a second image from a second imaging sensor, wherein the second imaging sensor faces the first imaging sensor; and

using a controller to place the first and second images in one computer file for simultaneous display on a display device.

50. The method of claim 49, wherein the first and second images are time-correlated.

51. The method of claim 49, further comprising:

placing patient information data from only one of the images in the computer file for simultaneous display with the images on the display device.