JP2018130537A - Method and apparatus used for endoscope with distance measuring function for object scaling - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and apparatus used for an endoscope with a distance measuring function for object scaling, and a method and apparatus for capturing images of a scene using a capsule device including a camera.SOLUTION: An image sequence is captured using the camera when the capsule device travels through a human gastrointestinal tract. In addition, structured-light images are captured using the camera by projecting structured light to one or more objects in a field of view of the camera when the capsule device travels through the human gastrointestinal tract. The structured-light images are interleaved with regular images. The distance information with respect to the camera associated with objects in the regular images is derived. Both the image sequence and the distance information are output. A method of determining the size of an object of interest utilizing the distance information is also disclosed. In another method, the distance information is used to scale an object or adjust intensity.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an endoscope for capturing an image of a human gastrointestinal tract for examination purposes, and more particularly, to an endoscope used for measuring the distance of an object in the field of view of a camera. The distance information is used to process the captured image sequence, for example, to measure the size of the object of interest or to stitch the video sequence to reduce the viewing time.

[Reference to related applications]
The present invention is a continuation application of PCT / US17 / 15668 of PCT international patent application dated January 30, 2017, and claimed the priority of the PCT international patent application. The present invention is also a continuation-in-part of US patent application Ser. No. 14 / 884,788, filed Oct. 16, 2015, and claimed the priority of the US patent application. The entire contents of the PCT international patent application and the US patent application are cited in the present invention.

  2. Description of the Related Art Conventionally, an imaging device that generates an image of a body cavity or passage of a living body including an endoscope and an autonomous encapsulated camera is known. An endoscope is a flexible or rigid pipe that is inserted into the body through a body opening or surgical opening and is typically passed through the mouth into the esophagus or through the rectum into the colon. An image is formed at the tip using a lens and transmitted to the proximal end outside the body by either a lens relay system or a coherent fiber-optic bundle. As a device that is conceptually similar, an image is electronically recorded at the tip using, for example, a CCD or CMOS array, and the image data is transferred as an electrical signal to the hand end via a cable. Endoscopes are widely used diagnostic tools because doctors can control their visual fields.

  The capsule endoscope is a different type of endoscope that has been developed in recent years. In a capsule endoscope, the camera can be swallowed along with a radio transmitter that transmits data, mainly containing images recorded by the digital camera, to an external base station receiver or transceiver and data recorder. Stored in a capsule. The capsule may include a wireless receiver for receiving instructions or other data from the base station. Instead of radio frequency transmission, low frequency electromagnetic signals may be used. Power may be supplied from an external inductor to the internal inductor of the capsule in an inductive manner, or may be supplied from a battery within the capsule.

  Patent application 1 entitled “In Vivo Autonomous Camera with On-Board Data Storage or Digital Wireless Transmission in Regulatory Approved Band” approved on July 19, 2011 includes on-board data storage. An autonomous capsule camera system is disclosed. The capsule camera with a built-in storage stores the acquired image in a non-volatile memory. The capsule is collected after it has left the human body. The collected image stored in the non-volatile memory of the capsule camera is accessed via the output port of the capsule camera.

  This endoscope, when imaging the gastrointestinal tract of the human body, is to identify any possible abnormalities as one primary purpose. If any anomaly is discovered, the goal of further interest is to determine the characteristics of the anomaly, for example the size of the anomaly. The captured image is subjected to a professional medical examination and used for examination or diagnosis. Generally, the number of captured images is 25000 or more. Even experts who are familiar with the technique need a long censorship time to determine the image. Therefore, image stitching is used to reduce the number of images to be censored. As an example, Patent Document 2 of PCT international patent application published on December 4, 2014 discloses a technique for stitching an image captured by a capsule camera. There is a need to develop a method or apparatus that can further improve the efficiency of image stitching.

US Patent No. 7,983,458 WO2014 / 193670 A2 Publication

  The present invention discloses a method and apparatus for capturing an image of a scene using a capsule camera. After the patient swallows the capsule camera, the capsule camera captures an image when the capsule camera is going through the gastrointestinal tract. Also, when the capsule camera is passing through the gastrointestinal tract, a structured-light image is captured. Both a regular image and the structured light image or derived information of the structured light image are output. Distance information relating to the object in the regular image for the capsule camera can be derived.

  Information related to the distance information and the corresponding image of the regular image is output. The related information corresponds to the number of frames or the number of captures of the corresponding image of the regular image.

  The present invention further discloses a method for determining the size of an object of interest in an image. A regular image captured by the capsule camera and the structured light image are received. The distance information for the object of the object in the regular image is derived from the structured light image. The size of the object of interest in the selected regular image is determined by the pixel data of the target object and the distance information. The size of the object of interest is determined by the image size of the object image of interest in the selected regular image scaled by a factor of the object distance to the capsule camera and the focal length of the capsule camera. . The size of the object image of interest is obtained by surveying based on the number of pixels of the object of interest in the selected regular image.

  The present invention further discloses a method of generating a stitched image sequence by stitching the regular image using information including the distance information. Further, the distance information can be derived from the structured light image. As an example, for stitching the regular image, the distance information is used to scale the object in the regular image. As another example, for stitching the image sequence, the distance information is used to adjust the image intensity of the regular image.

FIG. 1 is a diagram illustrating the relationship among the object size, the size of the corresponding object image, the object distance, and the focal length of the camera. FIG. 2A is a diagram illustrating different sizes of object images of the same object in two images captured at two different object distances. FIG. 2B is a diagram illustrating different sizes of object images of the same object in two images captured at two different object distances. FIG. 3 is a diagram illustrating a flowchart for capturing a regular image and a structured light image according to an embodiment of the present invention, and the structured light image shows distance information about an object in the regular image for the camera. To derive. FIG. 4 is a diagram illustrating a flowchart for determining the size of an object of interest in a regular image based on pixel data of the object and the distance information according to an embodiment of the present invention. FIG. 5 is a diagram illustrating stitching a regular image using information including the distance information to generate a stitched image sequence according to an embodiment of the present invention.

  It will be readily appreciated that the elements of the present invention may be installed and designed utilizing a variety of different aspects, as described or illustrated in the specification and drawings. Thus, as shown in the figures, the following detailed description of the embodiments of the invention does not limit the scope of the invention claimed in the claims, but represents a preferred embodiment of the invention. It's just a thing. As used throughout this specification, "one embodiment", "embodiment", or similar terminology includes a particular feature, structure, or characteristic described with respect to the embodiment in at least one embodiment of the invention. Means that Thus, the terms “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily referring to the same embodiment.

  Also, in one or many embodiments, the described features, structures, or characteristics may be combined in any suitable manner. However, it will be apparent to those skilled in the art that the present invention may be practiced with one or more specific details omitted, or by other methods or components, etc. It is. In other instances, well-known structures and operations are not shown or described in detail to avoid confusing aspects of the invention. Embodiments shown in the present invention can be best understood with reference to the drawings, wherein like reference numerals are used to refer to like members throughout. The following description is merely illustrative and merely illustrates selected embodiments consistent with the claimed apparatus and method according to the claims.

  In general, an endoscope is inserted into a human body via a natural opening such as an oral cavity or an anus. For this reason, the endoscope is preferably small in size so as to minimize intrusion. As described above, the endoscope can be used for diagnosis of the human gastrointestinal (GI) tract. By observing the captured image sequence, any possible anomalies can be identified. If any anomalies are discovered, the next purpose is to identify the characteristics of the anomalies, such as size. Accordingly, the present disclosure is an endoscope that includes a distance surveying device that measures object distances between the camera and different positions of objects in the field of view of the camera.

  There are various types of devices that measure the object distance between the camera and different positions of the object in the field of view of the camera. As an example, one of the distance surveying devices determines the distance based on time of flight (ToF, Time of Flight) or phase shift of the light source. The light source may be a laser or a light emitting diode (LED). The optical sensor detects the reflected light. The distance is determined from the time difference or phase difference between the light emitted from the light source and the light incident on the optical sensor. Ultrasound is also a signal source applicable to surveying the distance between an object and the camera (used for gastrointestinal imaging). The distance surveying apparatus is conventional, and various documents are known that describe the distance surveying based on ToF or phase shift using light or ultrasonic waves. For this reason, details of the distance surveying device based on ToF or phase shift using light or ultrasonic waves are omitted in the present invention.

  If the light source measures distance, the light for measuring the distance interferes with the flash that illuminates the gastrointestinal tract during the image capture period. In this case, it is necessary that the light for measuring the distance and the flash for capturing the image are not applied simultaneously, or at least one light source is substantially darkened. The distance information may be stored individually or together with related images. The distance information may be captured before or after the related image. When the distance information is stored separately, the related information of the related image (related information in the present invention) is also stored, and the distance information can be used appropriately. The relationship information may be the time when the relationship image is captured, the frame time, or the number of frames. If the distance is measured by the ultrasonic wave, the distance measurement by the ultrasonic wave can be generated simultaneously with the image capture of the gastrointestinal tract by the application of the flashlight.

  Although the use of distance surveying devices based on ToF or phase shift using light or ultrasound is known, it is difficult and expensive to install such a distance surveying device in an endoscope because it is sized This is because of its large size and inappropriate application to an endoscope. Therefore, other distance surveying devices perform image processing using images captured using an image sensor.

  As an example, as one technique for obtaining depth information, a color filter having an appropriately narrow passband and simultaneously obtaining color information and depth information is used, which is provided at the top of a selected sensor. . A negligible energy is projected onto the sensor by an ambient light source having a frequency spectrum in the passband of the color filter. In the RGB pixel, a fourth type of pixel for capturing light having a frequency spectrum in a pass band of a filter provided at the top of the pixels may be added. Then, structured light having substantially a frequency spectrum in the pass band can be projected onto the scene. However, this means reduces the spatial identification rate of the image or video captured by the image sensor, so a non-traditional color filter is required.

  Another technique obtains depth information and 3D topology from structured light images that can be viewed with an RGB sensor. However, real-time images and / or videos are mixed into the structured light image provided thereon. When a structured light image is captured, information about the depth or shape of the object in the scene can be derived. The depth or shape information is applied to an image captured immediately before or after the image or the structured light image. Also, since the regular image was captured with a capsule endoscope at a very slow frame rate (eg, 5 frames / second), the scene corresponds to the image captured with the structured light, and The scene corresponds to a regular image, and the images are clearly different due to movement of the endoscope or peristalsis of the intestines. US patent application Ser. No. 14 / 884,788, filed Oct. 16, 2015, includes a structured structure with a shortened frame period to improve the accuracy of depth information derived from the structured light image. An optical image is disclosed. In time, the structured light image is close to the regular image, so the depth information derived thereby is compared to the information derived from the structured light image having a longer frame period. And accurate.

  When using a distance surveying device based on structured light, the depth (ie, distance) information is derived from the structured light image. That is, unprocessed distance information is stored in the form of a structured light image. In this case, the distance information (that is, the structured light image) may be stored separately, or may be stored together with a related image acquired by regular light. The distance information can be acquired before or after the related image. If the distance information is stored separately, the related information (that is, related information) of the related image is also stored, so that the distance information can be used as appropriate. In this field, a technique for deriving the depth information from the structured light image is known. In the present disclosure, details for deriving the depth information from the structured light image are omitted.

  In an endoscope, the focal length is known by design. If the distance between the object and the camera (in this invention, the object distance) can be determined, the size of the object can be determined only by geometry. FIG. 1 shows a simple example of determining the size of an object based on the distance from the object to the camera. In the camera system, the image sensor is provided on a focal plane 120 behind the lens 110. The camera acquires a scene with an angle α extending from the field of view. The focal length f is the distance between the lens and the image sensor. In general, the application of the focal length to an endoscope is constant and is known by design. However, when the capsule endoscope passes through the gastrointestinal tract, the distance D of the object changes depending on the position of the capsule endoscope and the relative angle of the imaged gastrointestinal tract. If the distance D is known, the size of the object can be determined from the captured image by measuring the size of the object image in the image. As an example, if the distance from the object 130 having the height H to the camera is D, the height H of the object image is derived from the following equation 1 based on the height h of the object image in the image. can do.

  In the above equation (1), h is measured from the image, the focal length f is known by design, and the distance D is obtained by determination by the selected distance surveying device. Therefore, if the distance can be determined, the size of the object can be derived. The actual size of the object in the image can be surveyed. However, since the image is acquired digitally, it is more convenient to describe the surveying of the size in number of pixels. And the actual size and optical footprint of the surface of the image sensor are known. Further, the number of pixels is known (for example, 320 × 240). For this reason, the size of the object image in the image can be measured by the number of pixels, and it can be converted into the size of the actual object image in the image.

  As described above, the size of the object image in the image is related to the actual object size and the distance to the camera. A small object at a close location appears to have the same size as a large object at a remote location. As an example, the object 140 is smaller than the object 130 but is closer in distance so that it appears to have the same height as the object 130 in the image. Therefore, the distance is important information on the size of the object. Therefore, the distance measuring device disclosed above determines the size of the object based on the image captured by the endoscope.

  The distance information is also effective for image stitching. In an ideal state of the actual object model, changes in the size of the object in the captured image are implicitly accommodated by a registration process. Corresponding objects in different images are identified and registered. In different images, different size assumptions of different changes are taken into account by the registration process. The objects having different sizes in the target image are matched to corresponding objects in the reference frame. Similarly, a global motion model is applied to the target image to scale the object to scale and stitch the image accordingly. However, in general, the images associated with objects in the gastrointestinal tract environment are quite different from the ideal actual object model. For example, when the optimization process extends to a variable of distance, the iterative process does not always converge or converge to a local minima. As is well known, it is normal for the iterative process to be used as part of the overall registration process. In one embodiment, the distance information is used for scaling. In particular, since the distance information contributes to the registration of the image, the accuracy of the registration can be improved.

  2A and 2B are diagrams illustrating different sizes of object images in two images obtained by capturing the same object at two different object distances. In FIG. 2A, illustration 210 corresponds to the situation where the capsule 211 is far from the object 213 of interest in the gastrointestinal tract 212. Image 220 corresponds to the image captured in example 210. In FIG. 2B, illustration 230 corresponds to the situation where the capsule 211 is close to the object of interest 213 in the gastrointestinal tract 212. Image 240 corresponds to the image captured in the example 230. Here, the image 240 is an image captured by a camera close to the wall of the gastrointestinal tract. For this reason, the object image in the image 240 appears to be larger than the object image in the image 220. Therefore, the distance information can be used for scaling the object in the two images.

  In the gastrointestinal tract environment, images are always captured with a light source that illuminates the field of view. When the camera approaches the tube wall of the gastrointestinal tract, the imaged object is bright and the intensity of the image is high. On the other hand, when the camera moves away from the tube wall of the gastrointestinal tract, the imaged object becomes dark and the intensity of the image decreases. For this reason, the intensity of the entire image is related to the distance between the object and the camera in the field of view. In the environment of the gastrointestinal tract, since the distance from the object to the camera is quite short, the change in the intensity of the entire image is quite large. Since such a large change in strength degrades the registration performance, the stitch performance is reduced.

  Therefore, in another embodiment of the present invention, the image intensity of two registered images is adjusted by the distance. Pixel intensity is inversely proportional to the square of distance or other functions. Instead of representing in the form of a function, the relationship between the intensity of the pixel and the distance can be represented in the form of a table. Adjustment can be made to reduce the intensity at a location close to the image to match the intensity at a location far from another image. Or it can adjust so that the intensity | strength in the location of a distant image may be improved, and it can match the intensity | strength in the location of another image. If the intensity adjustment compensates for changes due to different distances, the registration and image stitching should be performed better.

  FIG. 3 is a diagram illustrating a flowchart for capturing a regular image and a structured light image according to an embodiment of the present invention, and the distance information is derived from the structured light image. In step 310, the capsule device is administered to a patient. In step 320, unstructured light is projected from the unstructured light source onto the scene in the camera's field of view. In step 330, when the capsule device passes through the gastrointestinal tract of the human body, the camera is used to capture a regular image formed in the common image plane of the camera. In step 340, structured light is projected from the structured light source onto a scene in the field of view of the camera. In step 350, when the capsule device passes through the gastrointestinal tract of the human body, the structured light image is captured using the camera, and the structured light image is interleaved with the regular image to obtain the regular image. Information on the distance of the object to the camera at is derived. In step 360, the structured light image or information derived from the structured light image is output. In step 370, the distance information is output. The distance information extracted from the structured light image is determined by individual distance information at one or more locations in the image or the field of view. This invention refers to including one or more locations in the image.

  FIG. 4 is a diagram illustrating a flowchart for determining the size of an object of interest in the image sequence based on the object image sequence and the distance information according to an embodiment of the present invention. In step 410, a regular image captured by the capsule device is received as the capsule device passes through the human gastrointestinal tract. In step 420, the distance information for the object in the regular image selected by the capsule camera is determined. In step 430, the size of the object of interest in the selected regular image is determined based on the pixel data of the target object and the distance information. In step 440, size information regarding the size of the object of interest is output.

  FIG. 5 is an exemplary flowchart for generating a stitched image sequence by stitching an image sequence using information including the distance information according to an embodiment of the present invention. In step 510, a normal image captured by the capsule device is received as the capsule device passes through the human gastrointestinal tract. In step 520, when the capsule device passes through the human gastrointestinal tract, the capsule camera determines the distance information for one or more objects in the regular image selected. In step 530, the information including the distance information is used to stitch the regular image to generate a stitched image sequence. In step 540, the stitched image sequence is output.

  The above allows one skilled in the art to practice the invention provided for the description of the particular application and its requirements. Since various modifications of the described embodiments will be apparent to those skilled in the art, the general principles defined in this disclosure can be applied to other embodiments. Accordingly, the present invention is not limited to those disclosed embodiments, but is consistent with the principles and to the greatest extent consistent with the novelty features disclosed herein. In the foregoing detailed description, different specific details have been shown in order to provide a general understanding of the invention. However, those skilled in the art will understand that the present invention can be implemented.

  The present invention may be implemented in other specific forms without departing from the spirit and essential characteristics thereof. The described example is intended to be illustrative and should not be considered limiting. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

  110 lens, 120 back focal plane, 130 object, 140 object, 210 example, 211 capsule, 212 gastrointestinal tract, 213 object of interest, 220 image, 230 example, 240 image, D object distance, D ′ object distance , F Focal length, H Object image height.

Claims (20)

In a method of capturing an image of a scene using a capsule device with a camera,
Administering the capsule device to a patient;
Projecting unstructured light from an unstructured light source onto a scene in the field of view of the camera;
When the capsule device passes through the gastrointestinal tract of a human body, the camera captures a regular image formed in a common image plane of the camera;
Projecting structured light from a structured light source onto the scene in the field of view of the camera;
As the capsule device passes through the gastrointestinal tract of the human body, the camera captures a structured light image formed in the common image plane of the camera and interleaves the structured light image with the regular image. Deriving distance information about the object in the regular image for the camera;
Outputting the regular image and outputting the structured light image or derived information of the structured light image;
A method for capturing an image of a scene using a capsule device with a camera.   The information regarding the distance information of the object in the regular image with respect to the camera is derived from the structured light image, and the derived information of the structured light image corresponds to the distance information. A method of capturing an image of a scene using a capsule device with a camera.   The method for capturing an image of a scene using the capsule device with a camera according to claim 2, wherein the related information between the distance information and the corresponding image of the regular image is output.   The method of capturing an image of a scene using the capsule device with a camera according to claim 3, wherein the related information corresponds to a frame number or a capture count of the corresponding image of the regular image. In an endoscope that captures an image of a scene using a capsule device,
The capsule device
A camera,
An unstructured light source;
A structured light source;
One or more processors coupled to the camera and the structured light source;
One or more output interfaces coupled to the one or more processors;
A housing capable of swallowing,
The one or more processors are:
When the capsule device passes through the human gastrointestinal tract, the camera is used to project unstructured light onto the unstructured light source to capture a regular image;
When the capsule device passes through the gastrointestinal tract of the human body, the structured light image is interleaved with the regular image by causing the structured light source to project structured light onto a scene in the field of view of the camera using the camera. To derive distance information about the object in the regular image for the camera,
Outputting the regular image by the one or more output interfaces;
Outputting the structured light image or derivation information of the structured light image;
An endoscope for capturing an image of a scene using a capsule device, wherein the housing is configured to seal the camera and the one or more processors in a sealed environment. The one or more processors further derives the distance information about the object in the regular image for the camera from the structured light image;
The endoscope for capturing an image of a scene using the capsule device according to claim 5, wherein the derived information of the structured light image corresponds to the distance information.   The one or more output interfaces provide information related to the distance information and a corresponding image of the regular image and output the related information, and capture an image of a scene using the capsule device according to claim 6. Endoscope.   The endoscope that captures an image of a scene using the capsule device according to claim 7, wherein the related information corresponds to the number of frames or the number of captures of the corresponding image of the regular image. In a method of processing an image captured by a capsule camera,
Receiving a regular image captured by the capsule camera when the capsule camera passes through the gastrointestinal tract of a human body;
Determining distance information of the target object in the selected regular image with respect to the capsule camera;
Determining the size of the target object in the selected regular image based on the pixel data of the target object and the distance information;
Outputting size information relating to the size of the target object,
The distance information is derived from a structured light image captured by the capsule camera as it passes through the gastrointestinal tract of the human body.
A method of processing images captured by a capsule camera.   The size of the target object is determined by the image size of the target object of the selected regular image scaled by the ratio of the object distance to the capsule camera and the focal length of the capsule camera. Item 10. A method for processing an image captured by a capsule camera according to Item 9.   The method of processing an image captured by a capsule camera according to claim 10, wherein the image size of the target object is measured by the number of pixels in the target object of the selected regular image.   Receiving the structured light image from the capsule camera and deriving the distance information from the structured light image, wherein the structured light image is structured into a scene in the field of view of the capsule camera using the capsule camera 10. A method of processing an image captured by a capsule camera according to claim 9, which is captured by projecting light. In a method of processing an image captured by a capsule camera,
Receiving a regular image captured by the capsule camera when the capsule camera passes through the gastrointestinal tract of a human body;
Determining distance information for the capsule camera of one or more target objects in the regular image;
Stitching the regular image with information including the distance information to generate a stitched image sequence;
Outputting the stitched image sequence;
The distance information is derived from a structured light image captured by the capsule camera as the capsule camera passes through the gastrointestinal tract of the human body. A method of processing an image captured by a capsule camera.   Receiving the structured light image from the capsule camera and deriving the distance information from the structured light image, wherein the structured light image is structured into a scene in the field of view of the capsule camera using the capsule camera 14. A method of processing an image captured with a capsule camera according to claim 13, which is captured by projecting light.   15. The capsule camera of claim 14, wherein the distance information is used to scale the one or more target objects in the regular image to stitch the regular image. how to.   15. The method of processing an image captured with a capsule camera according to claim 14, wherein the distance information is used to adjust the image intensity of at least one of the regular images to stitch the regular image.   14. The method of processing an image captured by a capsule camera according to claim 13, wherein information related to the distance information and a corresponding image of the regular image is also received and used for stitching the regular image. An apparatus for processing an image captured by a capsule camera,
The apparatus includes one or more electronic circuits or processors,
When the capsule camera passes through the human gastrointestinal tract, it receives a regular image captured by the capsule camera,
Determining distance information for one or more target objects in the regular image relative to the capsule camera, the distance information being structured by the capsule camera as it passes through the gastrointestinal tract of the human body Derived from light images,
Stitching the regular image using the information including the distance information to generate a stitched image sequence;
Configured to output the stitched image sequence;
A device that processes images captured by a capsule camera.   The one or more electronic circuits or processors receive the structured light image from the capsule camera and derive the distance information from the structured light image, the structured light image using the capsule camera The apparatus for processing an image captured by a capsule camera according to claim 18 captured by projecting structured light onto a scene in the field of view of the capsule camera.   The apparatus for processing an image captured by a capsule camera according to claim 18, wherein information related to the distance information and a corresponding image of the regular image is also received and used for stitching the regular image.