Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B6/00—Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10056—Microscopic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing

Definitions

• the present invention relates to a method and system for super-resolution of confocal images acquired through an image guide, as well as to a device used for the implementation of such a method.
• the invention is of a broader scope since it can be applied to any field in which imaging is carried out by means of a guide composed of a plurality of optical fibers, such as for example in the area of observation of the interior of a manufactured device.
• the image guide provides an image.
• a device allows the laser scanning, the light source and the receiver to be moved away from the object to be observed.
• the image guide is an assembly of several thousands of optical fibers whose spatial arrangement is identical at input and at output.
• the distal end that is to say close to the object to be observed therefore far from the light source, is associated with a head made up of optics in order to focus the laser beam in the object to be observed.
• a head made up of optics in order to focus the laser beam in the object to be observed.
• Such an image guide makes it possible to observe the object in depth, with a lateral resolution and an observation field which depend on the magnification of the optics, the inter-core distance of the image guide and the diameter of the guide. By changing the magnification, you can vary the resolution, to the detriment of the field. In the same way, a similar image guide, but with a smaller inter-core distance makes it possible to obtain the same results. In both cases, when the resolution is lower, the image field also decreases.
• An image guide is a fixed structure whose ratio between the useful surface and the number of cores present defines the resolution of the system.
• the inter-core distance between fibers cannot be reduced due to physical and technological constraints.
• manufacturing constraints come into play.
• physical constraints linked to the crosstalk of the guide, and to the optical properties of the fibers making it possible to guide the light around the visible wavelengths.
• the physical limits of guidance by bundle of optical fibers do not make it possible to obtain a better resolution with magnification and with fixed field.
• the present invention aims to increase the resolution of an image acquired through an image guide.
• Another remarkable object of the invention is to apply the concept of super-resolution to an image guide.
• the invention also aims to improve the resolution of a constant field image acquired through an image guide by increasing the number of measurement points per unit area.
• At least one of the above-mentioned objectives is achieved with a method for increasing the resolution of confocal images acquired through an image guide made up of a plurality of optical fibers, the proximal end of this image guide being connected a laser scanning device designed to emit a laser beam in each optical fiber of the image guide and collect each return beam during an acquisition, the distal end being associated with an optical head for focusing the laser beam emitted by the image guide in an observation object.
• this method comprises the steps of: - making a plurality of acquisitions through the image guide, each acquisition being carried out for a given spatial offset from said end distal of the image guide relative to the optical head,
• a redundant acquisition of the same object is carried out before a step of reconstructing the final image.
• the point cloud registration step is an abuse of language to mean that in fact each point cloud is processed so as to readjust the images corresponding to these point clouds, without necessarily reconstructing these images.
• a single acquisition through the image guide is equivalent to an acquisition on an irregular grid (which is the location of the fibers), called in the following sampling of the object. Moving the image guide in the tube is like moving the sampling scheme which is the arrangement of the individual fibers in the image guide.
• the step of transforming the acquired data can be followed by the application of a filter to eliminate artifacts due to the presence of the image guide.
• This filter can consist of a method of image processing acquired by means of a guide constituted by a plurality of optical fibers. More precisely, for each optical fiber, an area corresponding to this optical fiber is isolated from the acquired image (the raw data), then the information from this area is used to estimate the power transmitted by this fiber, and the power coming from the object observed via this same fiber.
• a representation of the object observed is then a weighted point cloud which can be reconstructed in the form of an image by interpolation of said weighted point cloud on a square grid.
• Such a transformation power estimation + interpolation eliminates the pattern due to optical fibers.
• the registration step comprises the following stages:
• the mean square error is calculated by considering only the pixels corresponding to the points of said mobile point clouds.
• the interpolation algorithm is for example a series of iterative B-spline approximations notably disclosed in the following document: Seungyong Lee,
• the cloud of reference points can correspond to a position of the image guide at rest.
• the registration stage can for example consist in registering each point cloud as a function of predetermined offset distances obtained by a stage for calibrating the displacements of the image guide.
• At each acquisition to offset the image guide relative to the optical head, at least one voltage is applied to at least one piezoelectric strip, which is at least integral with the distal end of this image guide .
• at least one voltage is applied to at least one piezoelectric strip, which is at least integral with the distal end of this image guide .
• four bands constituting a piezoelectric tube which surrounds at least the distal end of the image guide are used, and a pair of voltages is applied for each displacement of the distal end of the image guide opposite on two opposite bands respectively.
• the spatial offset is obtained by a substantially lateral translation movement of the distal end of the image guide in two orthogonal or at least non-collinear directions.
• the calibration step can be carried out by applying the following steps to a limited number of point clouds obtained by acquiring a reference pattern placed in place of said observation object: - correction of geometric distortions,
• the calibration makes it possible to validate a linear displacement model as a function of the applied voltage and to estimate an offset coefficient as a function of the measurements.
• the subsequent point clouds are then readjusted using this linear model.
• a device for increasing the resolution of confocal images acquired through an image guide made up of a plurality of optical fibers, the proximal end of this image guide being connected to a laser scanning device designed to emit a laser beam in each optical fiber of the image guide and collect each return beam during an acquisition, the distal end being associated with an optical head for focusing the emitted laser beam by the image guide in an observation object.
• the optical head comprises optical means integral with this optical head.
• the device further comprises a piezoelectric tube surrounding the image guide and integral with this image guide, at least at said distal end, so as to spatially offset this distal end relative to the optical head in response to a set point. offset. This setpoint is preferably a pair of voltages applied to the piezoelectric tube.
• the piezoelectric tube advantageously consists of at least four independent ceramic bands each occupying a quarter of the tube.
• the internal and external faces of each strip can be covered with metallic material, such as silver, so that the application of opposite tensions on two opposite strips respectively, shifts the distal end of the tube using the transverse piezoelectric effect .
• the ceramic bands can be ordered two by two in two orthogonal directions.
• a system for increasing the resolution of confocal images acquired through an image guide made up of a plurality of optical fibers, the proximal end of this image guide being connected to a laser scanning device. intended to emit a laser beam in each optical fiber of the image guide and collect each return beam during an acquisition, the distal end being associated with an optical head for focusing the laser beam emitted by the image guide in an object of observation.
• this system comprises:
• - Figure 2 is a schematic view of the piezoelectric tube surrounding the image guide
• - Figure 3 is a schematic view illustrating the movement of the end of the image guide
• FIG. 4 is a schematic view illustrating an irregular superposition of point clouds
• FIG. 5 is a block diagram illustrating the principle of registration of point clouds.
• FIG. 1 we see an ordered bundle of flexible optical fibers (in particular several tens of thousands) forming an image guide 1 with, on its proximal end, a light source 2 and a fiber injection system making it possible to illuminate the fibers one by one and, on its distal end, an optical head 3 making it possible to focus the beam leaving the illuminated fiber at a point situated at a given depth of the object observed 4.
• the injection system comprises several optical elements 5 preceded by a fiber scanning system 6, such as a diverter, making it possible to scan the fibers one by one at very high speed. Each fiber is used in turn to convey the illumination beam and also the corresponding return beam from the object observed.
• the spatial resolution is obtained by focusing the laser beam at a point and by the confocal character residing in the spatial filtering of the object observed by the same fibers as those used for the illumination. This makes it possible to receive, by means of a photodetector 9, exclusively the signal coming from the object observed and to produce an image point by point.
• the distal end of the image guide is inserted into a piezoelectric tube 7, itself mounted in a rigid tube 3 forming an optical head and containing optics 8 placed at the output of the image guide 1.
• the piezoelectric tube 7 makes it possible to move the image guide 1 inside the optical head 3 which remains fixed relative to the object observed. This avoids the problems due to the friction of the optical head on the object observed.
• FIG. 2 the configuration of the piezoelectric tube 7 is seen in perspective.
• This tube 7 consists of four ceramics each occupying a quarter of cylinder.
• the tube 7 is coated with silver on the internal and external faces in order to use the transverse piezoelectric effect.
• This phenomenon explains the deformation of a crystal when it is immersed in an electric field, it results from the existence of electric dipoles in the crystal configuration.
• a tension is applied between the internal and external face of a ceramic, this one lengthens (or shrinks according to the sign of the tension) and becomes thinner (respectively wider).
• the scanning is therefore done by applying a positive voltage on one of the ceramics and the opposite voltage on the opposite ceramic.
• FIG. 3 illustrates a displacement of the piezoelectric tube 7 along an axis.
• the solid lines represent the rest position, without offset.
• the dotted lines represent the end of the piezoelectric tube 7 offset upwards, thus causing the offset of the distal end of the image guide 1.
• the optics 8a and 8b, integral with the optical head, as well as the object observed 4 remain fixed.
• the displacement of the distal end of the image guide corresponds to a displacement of the point of impact of the laser beam on the object observed.
• the super-resolution method according to the invention comprises a first step in which for a given fixed position of the optical head, a series of acquisitions is carried out. For each acquisition, the laser beam scans all of the optical fibers. The photodetector 9 then recovers a set of raw data which can be represented by arranging them on a grid according to the order of the laser scanning. We see appearing for each fiber domes representative of the position of the fibers in the guide with respect to laser scanning. The offset of the distal end of the image guide 1 does not change the arrangement or the position of the optical fibers at the proximal end of the image guide. On the other hand, the object observed through these domes similar to a grid with holes moved when the piezoelectric tube 7 was deformed.
• each fiber an area of influence that can be defined as the width of the optical transfer function of each fiber combined with that of the optics placed between the output of the image guide and the object observed.
• the transfer function of a fiber is not necessarily equal to that of the neighboring fiber. In practice they can be considered as being all different, at least in their width.
• the point clouds Before the point clouds can be superimposed, they must be reset. Indeed, we acquired the same object, but the sampling scheme is shifted. After correcting the geometric distortions of the system, the offset is reduced to a translation which is evaluated by comparing all the acquisitions of the same object with a reference acquisition, in general that where the piezoelectric tube 7 is at rest (by convention).
• the problem of registration of images is a fairly classic subject and many techniques exist.
• the problem consists in resetting two images of the same object taken before and after shifting thanks to the piezoelectric mechanism.
• the underlying transformation model is a translation model.
• the number of parameters to be estimated is therefore only two (x, y).
• the transformation model of the present case differs from many studies because it uses an irregular grid.
• the best transformation is sought by minimizing a cost function with two parameters (the values of the translation along the axes of translation).
• two parameters the values of the translation along the axes of translation.
• FIG. 5 The principle of the algorithm retained is illustrated in FIG. 5.
• a reference point cloud hereinafter referred to as a fixed point cloud.
• We will then interpolate this fixed point cloud so as to reconstruct an image hereinafter designated as a fixed image 10.
• the interpolation of the fixed point cloud to obtain the fixed image is carried out using the iterative b-spline approximation algorithm disclosed in the document by Seungyong Lee et al., which is briefly described little further.
• the interpolation of the fixed point cloud to obtain the fixed image is carried out using the iterative b-spline approximation algorithm disclosed in the document by Seungyong Lee et al., which is briefly described little further.
• each other point cloud hereinafter referred to as a mobile point cloud
• a mobile point cloud is interpolated so as to reconstruct a mobile image.
• the advantage is a considerable time saving.
• An interpolation flap 12 of the transformation in which the initial mobile point cloud is translated by the transformation in progress;
• a distance calculating step 13 in which a distance is measured between the moving image from the translated point cloud 11 and the fixed image
• This distance is a mean square error evaluated between the points of the mobile point cloud and the interpolated still image
• a gradient-based optimizer 15 evaluates the local variations in the distance previously calculated in order to find a new transformation (ie translation) which makes it possible to decrease the value of the distance (that is to say optimize it) at the next loop.
• a limited number of possible offsets are made, for example 64, and they are readjusted with respect to a common reference, for example a point cloud obtained when the piezoelectric tube is at rest. Then a linear regression is carried out on all of the voltage pairs applied to the piezoelectric - offset measured by registration. If this model is linear and has a satisfactory accuracy (much less than the inter-core distance), we consider that the model is valid, and we can use it directly thereafter on subsequent acquisitions made on the measurement object , without having to re-register the images represented by the point clouds.
• This preferred embodiment therefore allows much faster processing since the point clouds acquired with the observed object do not undergo the whole registration process with interpolation. This process is only carried out for 64 point clouds acquired with a test pattern or other.
• f be the function to be reconstructed.
• f in the form of a uniform bicubic B-spline function defined on a rectangular mesh maill covering the support of f.
• the function obtained is therefore C 2 .
• ⁇ y the values associated with the nodes of the mesh. Without loss of generality, we suppose that this mesh is composed of the points of whole coordinates on a rectangle of the plane.
• the B-spline is given by:
• the control points ⁇ y are estimated by iterative approximations, using hierarchical lattices. We go from one scale to the next by refining the trellis by a factor of 2. Each point in the sampling will influence 16 control points. Without inverting the whole system, we will at each scale calculate the contributions of the previous scales locally, and deduce the residuals to be estimated for this scale. Each sampling point will influence the nearest 16 control points. This algorithm makes it possible to obtain an interpolation at the convergence of the algorithm (finite number of iterations). In the present case, two borderline cases can be considered for the size of the trellis. The first is the case where there is not a single control point ⁇ y which is not influenced by at least one point of the sampling.
• the second is the case where any two points in the sampling cannot influence the same control point.
• These two cases make it possible to calculate the size of the starting trellis, and the size of the trellis making it possible to reach convergence.
• These sizes can be calculated empirically assuming that the distribution of points is that of a hexagonal sampling. We observe the histogram of the distances between the neighboring sampling points (in the sense of their Voronoi diagram), and we keep the quantile at 5% (or any other small percentage). This minimum distance is in fact the distance between two offsets.
• a step of deconvolution of the signal by making the approximation that the system is linear invariant, and by taking a function of average transfer of a fiber of the image guide.
• Wiener filtering for example can be used.

Landscapes

• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Optics & Photonics (AREA)
• Microscoopes, Condenser (AREA)
• Mechanical Optical Scanning Systems (AREA)
• Image Processing (AREA)
• Image Input (AREA)
• Endoscopes (AREA)
• Image Analysis (AREA)

Priority Applications (10)

Applications Claiming Priority (2)

Publications (1)

Family

ID=34639737

Family Applications (1)

Country Status (13)

Cited By (2)

Families Citing this family (35)

Citations (4)

Family Cites Families (13)

• 2003
  • 2003-12-31 FR FR0315628A patent/FR2864631B1/fr not_active Expired - Fee Related
• 2004
  • 2004-12-29 BR BRPI0418235-9A patent/BRPI0418235A/pt not_active Application Discontinuation
  • 2004-12-29 CN CN2004800394794A patent/CN1902660B/zh not_active Expired - Fee Related
  • 2004-12-29 WO PCT/FR2004/003401 patent/WO2005073912A1/fr not_active Ceased
  • 2004-12-29 ES ES04817612T patent/ES2383526T3/es not_active Expired - Lifetime
  • 2004-12-29 US US10/585,083 patent/US7646938B2/en active Active
  • 2004-12-29 JP JP2006546264A patent/JP5268257B2/ja not_active Expired - Lifetime
  • 2004-12-29 AT AT04817612T patent/ATE547772T1/de active
  • 2004-12-29 EP EP04817612A patent/EP1702304B1/fr not_active Expired - Lifetime
  • 2004-12-29 AU AU2004314774A patent/AU2004314774B2/en not_active Ceased
  • 2004-12-29 CA CA2550305A patent/CA2550305C/fr not_active Expired - Lifetime
  • 2004-12-29 KR KR1020067015382A patent/KR20060131828A/ko not_active Withdrawn
• 2006
  • 2006-06-18 IL IL176372A patent/IL176372A/en active IP Right Grant

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events