WO2023045116A1 - Procédé de construction d'image - Google Patents

Procédé de construction d'image Download PDF

Info

Publication number
WO2023045116A1
WO2023045116A1 PCT/CN2021/138107 CN2021138107W WO2023045116A1 WO 2023045116 A1 WO2023045116 A1 WO 2023045116A1 CN 2021138107 W CN2021138107 W CN 2021138107W WO 2023045116 A1 WO2023045116 A1 WO 2023045116A1
Authority
WO
WIPO (PCT)
Prior art keywords
images
image
imaging
target
splitting device
Prior art date
Application number
PCT/CN2021/138107
Other languages
English (en)
Chinese (zh)
Inventor
李剑平
陈涛
马文齐
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Publication of WO2023045116A1 publication Critical patent/WO2023045116A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the invention relates to the technical field of photoelectric imaging, in particular to an image construction method.
  • optical parameters of an optical imaging system include optical resolution, depth of field, dynamic range, color state, polarization state, and spectral state, etc. These properties play a vital role in the practical application of the imaging system.
  • optical imaging method itself or the working principle of the photosensitive device, it is difficult to achieve a high level of the above optical parameters at the same time, such as:
  • the imaging system must sacrifice a certain optical resolution and sensitivity to perform color imaging
  • the imaging system sacrifices the diversity of imaging targets to achieve high dynamic range imaging of a certain type of target.
  • the Chinese invention patent application document with the application publication number CN113052766A discloses a multi-scale imaging device, a large field of view and high resolution image stitching method and system, in which for the gap between the large field of view and high resolution Imaging Contradiction proposes a solution to achieve the effect of coordinating the two by splicing images.
  • deep learning technology usually has higher requirements on the data set, not only the number of image pairs in the data set is large, but also the diversity is good.
  • the application of deep learning technology in the enhancement of optical parameters of imaging systems is still relatively small.
  • the present invention proposes an image construction method, which can construct image pairs or image clusters or video pairs of various optical parameters, satisfying the data set of deep learning Require.
  • the technical solution adopted in the present invention is an image construction method, the steps of which include:
  • the information of the observed sample is divided into multiple beams of signal light through the beam splitting device, and each beam of light emitted by the beam splitting device passes through a different imaging unit to form a target image containing the information;
  • Construct paired images or clustered images change the optical parameters of the different beams emitted by the beam splitting device when they are imaged, so that different imaging units simultaneously form target images with different optical parameters, and obtain paired images or clustered images of the same observation sample image;
  • Image registration processing align multiple images of the same observation target to obtain image pairs or image clusters or video pairs for each observation target;
  • the optical parameters at least include at least one of optical resolution, defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • the beam splitting device emits two beams, and the two beams respectively pass through an imaging unit to form an image containing the information, and the optical parameters are optical resolution, defocus distance, brightness, polarization state, color, spectrum , one of fluorescence.
  • the beam splitting device emits three or more light beams, and each beam of light emitted by the beam splitting device passes through an imaging unit to form an image containing the information, and the optical parameters include at least optical resolution, One or more of defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • At least one pair of imaging units adopts lenses of the same magnification, one of which adopts a color camera and the other adopts a monochrome camera, adjusting the two imaging units
  • the focal planes of the units are coincident and imaged at the same time.
  • At least one pair of imaging units adopts lenses with the same magnification, and one or more filters are inserted between one of the imaging units and the beam splitting device and select different wavelength bands of light beams to enter the imaging unit for imaging, and then adjust the two imaging units to image clearly, and then image at the same time.
  • At least one pair of imaging units adopts lenses with the same magnification, adjusts the exposure time and gain of one of the imaging unit cameras, and then adjusts the imaging of both imaging units to be clear After that, image simultaneously.
  • At least one pair of imaging units adopts lenses with different magnifications, and then adjust the focal planes of the two imaging units to overlap, adjust the depth of field to the illumination range of the observed sample Inside, imaging at the same time.
  • At least one pair of imaging units adopts lenses with the same magnification, and the focal planes of the two imaging units are adjusted to coincide, and one of the imaging units is fixed.
  • the other imaging unit moves along the optical axis direction of the light beam, and the two imaging units simultaneously image images at different positions.
  • At least one pair of imaging units uses lenses with the same magnification, and adjusts the light beam that irradiates the observation sample to monochromatic light, and replaces the beam splitting device It is a dichroic mirror, after adjusting the two imaging units to image clearly, they image simultaneously.
  • the light beam irradiating the observation sample is allowed to pass through a polarizer first, and one of the beams emitted by the beam splitting device is then allowed to pass through a polarizer, wherein the other One light beam does not pass through the polarizer, and the two light beams are simultaneously imaged by two different imaging units.
  • the information of the observed sample is divided into multiple beams of signal light by the beam splitting device and emitted, specifically: the illuminator is used to emit a beam of light, and after the beam is irradiated on the observed sample, it is scattered to the beam splitting device .
  • the illuminator is located at one side of the observation container, and the light beam of the illuminator is emitted along the optical axis of the incident light of the beam splitting device.
  • the illuminator is arranged on one side of the observation container, and the angle between the light beam of the illuminator and the optical axis of the incident light of the beam splitting device is greater than 90°.
  • the illuminator is arranged between the observation container and the beam splitting device, and the angle between the light beam of the illuminator and the optical axis of the incident light of the beam splitting device is less than 90°.
  • the illuminator is arranged around the observation container, and the light beam emitted by the illuminator is perpendicular to the optical axis of the incident light of the beam splitting device.
  • the devices used when constructing pairs of images or clustered images include:
  • a sample observation device which sends out sample information in the form of an optical signal
  • a beam splitting device which receives the light beam containing the optical signal and splits it into multiple beams with different paths;
  • An imaging device which includes at least two imaging units, and respectively receives at least one of the beams containing optical signals emitted by the beam splitting device for simultaneous imaging to obtain paired images or clustered images of the same observation sample;
  • An adjustment device which is arranged at the imaging unit, and/or between the imaging unit and the beam splitting device, to adjust the optical parameters of the imaging unit;
  • the optical parameters at least include at least one of optical resolution, defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • the adjustment device includes at least one of the following structures:
  • a position parameter adjustment structure arranged at the imaging unit and/or the beam splitting device, to adjust the distance between each of the imaging units and the beam splitting device;
  • An optical element adjustment structure arranged between the imaging device and the beam splitting device and/or between the illuminator and the sample observation device, to adjust the polarization state and/or wavelength band and/or brightness of the light beam;
  • An imaging unit adjustment structure the imaging unit includes a camera and a lens/multiple different lenses, and the imaging unit adjustment mechanism is used to switch the positions of multiple lenses in an imaging unit;
  • the beam splitting device adjustment structure the beam splitting device is one or more three-dimensional beam splitters or beam splitters or dichroic mirrors, the beam splitting device adjusting structure is used to adjust the number or type of the beam splitting device.
  • the position parameter adjustment structure includes an adjustment track arranged along the optical axis direction of the outgoing beam of the beam splitting device, and each of the imaging units is slidably fitted on the adjustment track to adjust the alignment with the beam splitter. distance between devices.
  • the optical element adjustment structure includes a mounting part, and the mounting part is detachably connected/replaceably connected to at least one of one or more polarizing plates, attenuating plates, and optical filters.
  • the image registration process includes:
  • aligning the target area whose judging result is clear with the target area contained in the other images in the paired images or clustered images refers to: extracting the coordinate information of the target area whose judging result is clear, and transforming the coordinate information to obtain According to the coordinate information of the same target area contained in the other images in the paired images or the clustered images, multiple target areas of the same observation target are cropped according to their respective coordinate information.
  • aligning the clear target area with the target area contained in the paired image or the remaining images in the clustered image refers to: using the clear target area as a template, and the paired image or clustered image Template matching is performed on other images in the image, and the area with the highest matching degree is the same target area, and image pairs or image clusters or video pairs are obtained by cropping and storing.
  • extracting the coordinate information of the target area whose judgment result is clear refers to: extracting the coordinates of the four vertices of the target area whose judgment result is clear, and transforming the coordinates of the four vertices to obtain paired images or clustered images. The coordinates of the four vertices corresponding to the target area in the remaining images.
  • the transformation of the coordinates of the four vertices is assisted by a registration model
  • the registration model includes a target
  • the target provides a plurality of plane/space features that can be detected or identified, and each The features are assigned coordinate information.
  • the target is a planar grid, and each intersection point of the grid can be detected or identified.
  • the target is a flat plate with a plurality of evenly distributed prisms protruding from it, and each corner point of the prisms can be detected or identified.
  • the present invention has the following beneficial effects:
  • the collected paired images or clustered images are aligned to form image pairs or image clusters or video pairs, which can meet the high requirements for images in deep learning, and then facilitate the direct construction of observation target dataset;
  • Fig. 1 is a schematic diagram of the system of the two-way imaging unit of the present invention.
  • Fig. 2 is a system schematic diagram of a three-way imaging unit of the present invention
  • Fig. 3 is the structural representation of observation vessel of the present invention.
  • Fig. 4 is a schematic diagram of a light source installation structure
  • Fig. 5 is a schematic diagram of the first light source installation method
  • Fig. 6 is a schematic diagram of the second light source installation method
  • Fig. 7 is a schematic diagram of a third light source installation method
  • Figure 8 is a front view of the first target
  • Fig. 9 is a side view of Fig. 8.
  • Fig. 10 is the front view of the second target
  • Figure 11 is a comparison diagram of plankton imaging at high magnification and low magnification
  • Figure 12 is a comparison diagram of high-resolution and low-resolution target imaging.
  • Image construction method the steps include:
  • the information of the observed sample is divided into multiple beams of signal light by the beam splitting device, and each beam of light emitted by the beam splitting device passes through a different imaging unit 7 to form a target image containing the information;
  • Constructing paired images or clustered images changing the optical parameters of the different beams emitted by the beam splitting device when they are imaged, so that different imaging units 7 simultaneously form target images with different optical parameters, and obtain paired images or clustered images of the same observation sample. cluster image;
  • Image registration processing align multiple images of the same observation target 3 to obtain an image pair or image cluster or video pair for each observation target 3;
  • the optical parameters at least include at least one of optical resolution, defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • each image contains images of multiple observation objects 3, and there is a certain deviation between each image, so it is impossible to directly form an image pair, image cluster, or video pair of the same observation object 3. Therefore, when constructing an image pair and image cluster of a certain observation target 3, it is still necessary to reprocess the imaged image, that is, after the above-mentioned image registration processing, the image pair or image corresponding to each observation target 3 can be obtained.
  • Image clusters or video pairs when constructing image pairs or image clusters or video pairs.
  • different optical parameters of the beams emitted by the beam splitting device can be changed to realize different image pairs or image clusters or video pairs of the same observation target 3 under different optical parameters, satisfying the depth Diversity requirements for learning technologies.
  • the imaging unit 7 When the imaging unit 7 performs imaging, different numbers of optical paths can be formed depending on the beam splitting device, that is, the number of beams emitted by the beam splitting device corresponds to the number of imaging units 7. Based on this, the actual imaging has the following conditions:
  • the beam splitting device emits two beams, as shown in Figure 1, the two beams respectively pass through an imaging unit 7 to form an image containing the information, the optical parameters are optical resolution, defocus distance, brightness, polarization state, color One of , spectrum and fluorescence;
  • the beam splitting device emits three or more beams of light, as shown in Figure 2, each beam of light emitted by the beam splitting device passes through an imaging unit 7 to form an image containing the information, and the optical parameters include at least optical resolution One or more of power, defocus distance, brightness, polarization state, color, spectrum, fluorescence.
  • the imaging pair or imaging cluster is obtained by simultaneously imaging the same observation target 3 under different optical parameters, when the beam splitting device only emits two beams, only two optical paths are formed, so only two imaging paths need to be set correspondingly
  • the unit 7 is enough, and the images formed by the two imaging units 7 can constitute a pair of images. Only when the beam splitting device emits three or more beams of light paths, can a plurality of imaging units 7 be set correspondingly, and then by changing the optical parameters of each imaging unit 7, the same observation target 3 can be formed under different optical parameters.
  • the different optical parameters can be the same optical parameter such as the numerical adjustment of the magnification, or it can refer to the simultaneous change of multiple optical parameters, such as changing the magnification of one beam and changing the magnification of the other beam
  • the change of the polarization state, the third channel can be used as the control group of the first two channels, so as to obtain a set of consistent image clusters.
  • optical parameters whether it is two optical paths or multiple optical paths, at least the following methods can be used when adjusting optical parameters:
  • At least one pair of imaging units 7 use lenses with the same magnification, one of which uses a color camera and the other uses a monochrome camera, adjust the two imaging units
  • the focal planes of unit 7 are coincident and imaged at the same time;
  • At least one pair of imaging units 7 should use lenses with the same magnification, and one or more optical filters should be inserted between one of the imaging units 7 and the beam splitting device , and select light beams of different wavelength bands to enter the imaging unit 7 for imaging, and then adjust the two imaging units 7 to image clearly and simultaneously image;
  • At least one pair of imaging units 7 should use lenses with the same magnification, adjust the exposure time and gain of one of the imaging unit 7 cameras, and then adjust the imaging of both imaging units 7 to be clear After that, imaging at the same time;
  • At least one pair of imaging units 7 should use lenses with different magnifications, and then adjust the focal planes of the two imaging units 7 to overlap and adjust the depth of field to within the illumination range of the observed sample , while imaging;
  • At least one pair of imaging units 7 adopts lenses with the same magnification, and the focal planes of the two imaging units 7 are adjusted to coincide, and one of the imaging units 7 is fixed.
  • Another imaging unit 7 moves along the optical axis direction of the light beam, and the two imaging units 7 image simultaneously at different positions;
  • At least one pair of imaging units 7 should use lenses with the same magnification, and adjust the light beam illuminating the observation sample to monochromatic light, and replace the beam splitting device with two
  • the dichroic mirror is used to adjust the imaging of the two imaging units 7 to form images at the same time;
  • the optical parameters to be adjusted are not limited to the adjustment methods listed above, and other optical parameters can also be appropriately adjusted by referring to the aforementioned methods, which will not be exhaustive in this embodiment.
  • plankton in order to collect images of plankton with higher contrast and more detailed information, the plankton in larger water bodies can be imaged, and at the same time, high signal-to-noise images can be obtained for smaller-sized plankton.
  • dark-field imaging can be used to collect data pairs with different defocus degrees and image pairs with different optical resolutions, so as to expand the depth of field of the imaging system through deep learning methods and improve the optical resolution at the same time. Imaging that satisfies high contrast, large spatial range and high spatial resolution;
  • reflective illumination can be used to construct bright fields with different defocus degrees and different optical resolutions. Image pairs or image clusters or video pairs, and then enhance the depth of field and optical resolution of the reflective brightfield imaging system through deep learning technology;
  • the fluorescence can be excited by laser grazing illumination, and at the same time, the reflected light can be used for bright field imaging. Imaging, constructing image pairs of bright-field microscopy images and fluorescence images, combined with deep learning technology, makes the bright-field microscopy imaging system have the ability of fluorescence imaging.
  • the information of the observed sample is divided into multiple beams of signal light by the beam splitting device, specifically, the illuminator is used to emit a beam of light, and after the beam is irradiated on the observed sample, it is then scattered to the beam splitter.
  • the arrangement of the illuminators is as follows:
  • the illuminator is located on one side of the observation container 1, as shown in FIG. 5, and the light beam of the illuminator is emitted along the optical axis of the incident light of the beam splitting device to form a bright field imaging mode;
  • the illuminator is arranged on one side of the observation container 1, as shown in FIG. 6, and the angle between the light beam of the illuminator and the optical axis of the incident light of the beam splitting device is greater than 90°, forming a dark field imaging mode;
  • the illuminator is arranged between the observation container 1 and the beam splitting device, as shown in Figure 7, and the angle between the light beam of the illuminator and the optical axis of the incident light of the beam splitting device is less than 90°, forming a reflective bright light. field imaging mode;
  • the illuminator is arranged around the observation container 1 , as shown in FIG. 1 , and the light beam emitted by the illuminator is perpendicular to the optical axis of the incident light of the beam splitting device.
  • the illuminator can illuminate the observation sample by various illumination methods, and according to the above-mentioned various adjustment methods, the required paired images or clustered images are formed in one or more imaging units 7, Due to the misalignment between each image, image registration processing is also required, including:
  • aligning the target area whose judging result is clear with the target areas contained in the other images in the paired images or clustered images refers to: extracting the coordinate information of the target area whose judging result is clear, and transforming the coordinate information to obtain the For the coordinate information of the same target area contained in the other images in the image or clustered images, multiple target areas of the same observation target 3 are cropped according to their respective coordinate information.
  • Extracting the coordinate information of the target area whose judgment result is clear refers to: extracting the coordinates of the four vertices of the target area whose judgment result is clear, and transforming the coordinates of the four vertices to obtain the paired images or the remaining images in the clustered images. The coordinates of the four vertices corresponding to the target area.
  • a registration model can be formed by making the target 6, as shown in Figure 8-10, the target 6 provides multiple plane/space features that can be detected or identified, and each of the features Assign coordinate information.
  • the target 6 can be a planar grid, that is, the coordinates of each intersection point of the grid are known; The coordinates of the corner points are all known, and the prism can be a triangular prism, a quadrangular prism or a multi-prism.
  • Another method for aligning the target area in which the target area that is judged to be clear is aligned with the target area contained in the other images in the paired image or clustered image, referring to: using the result
  • the clear target area is used as a template, and template matching is performed with other images in the paired image or clustered image.
  • the area with the highest matching degree is the same target area, and the image pair or image cluster or video pair is obtained by cropping and storing.
  • the image pair or image cluster or video pair of the same observation target 3 can be obtained, and different optical parameters can be changed to meet the sample diversity requirements of deep learning technology, and at the same time meet the requirements in terms of quantity.
  • Learning technology provides support for data set construction. Compared with the imaging of the observation target 3 in the prior art, this solution is also applicable to the observation target 3 in motion, with a wider application range and higher imaging precision.
  • Embodiment 1 an apparatus for constructing image pairs or image clusters or video pairs, as shown in Figure 1-2, includes:
  • a sample observation device which sends out sample information in the form of an optical signal
  • a beam splitting device which receives the light beam containing the optical signal and splits it into multiple beams with different paths;
  • An imaging device which includes at least two imaging units 7, and respectively receives at least one of the light beams containing optical signals emitted by the beam splitting device for simultaneous imaging to obtain paired images or clustered images of the same observation sample;
  • An adjustment device which is arranged at the imaging unit 7, and/or, between the imaging unit 7 and the beam splitting device, to adjust the optical parameters of the imaging unit 7;
  • the optical parameters at least include at least one of optical resolution, defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • the observation sample is a liquid containing plankton
  • the observation target 3 is a single plankton, as shown in Figure 1-4
  • the observation container 1 is a hollow cuboid, and one side extends horizontally outwards with a prominent observation Part 11, the observation part 11 is also hollowly arranged and communicated with the cuboid body part of the observation container 1.
  • the observation part 11 can be a cuboid or a cylinder, and an illuminator is installed on it.
  • the illuminator is a ring lamp.
  • the upper part of the observation container 1 is connected with a storage device for storing the sample solution through a pipeline, so as to replenish liquid in time
  • the bottom of the observation container 1 is also connected to the liquid discharge device through a pipeline
  • the liquid discharge pipeline is provided with a valve for controlling opening and closing, In order to discharge the sample solution after observation.
  • the draining pipeline is connected with an air pump.
  • the air pump starts, not only the observed sample solution in the observation container 1 can be discharged cleanly, but also the inside of the observation container 1 can be pumped into a negative pressure to realize the sample solution.
  • Automatic liquid feeding reducing manual operation.
  • the effect of automatic imaging and automatic replacement of observation samples can be realized.
  • a horizontal partition is also installed in the middle of the observation container 1.
  • plate 13 the bottom of the partition 13 should be level with the bottom of the observation part 11 , and the drain pipe passes through the partition 13 to communicate with the outside.
  • the target 6 also needs to be installed, so an installation groove 12 is formed on the upper surface of the horizontally extending observation part 11, and the opening direction of the installation groove 12 is in line with the extension direction of the observation part 11. vertical.
  • the entire observation container 1 is made of highly transparent material, such as acrylic, quartz glass and the like.
  • the illuminator in this embodiment is an annular light source 2, as shown in FIG. 4, which may be an LED aperture, which is sleeved on the observation part 11 to form an annular illumination.
  • a light source 2 adjustment frame and a controller are provided. The controller is used to control the light source 2 to turn on and off, and the light source 2 adjustment frame is used to fix the position of the light source 2 and adjust the position according to requirements.
  • the light source 2 adjustment frame includes two telescopic uprights 5, and the bottom of the uprights 5 is installed and connected with the fixed surface by bolts, so that the light source 2 can adjust the height, and also can adjust the installation position with the fixed surface.
  • the upper end of the column 5 of the light source 2 adjustment frame is also fixed with a horizontal connecting rod 51, the connecting rod 51 and the light source 2 are detachably connected by bolts, and the connecting rod 51 is a horizontal waist-shaped hole for the bolts to pass through, so that The position between the light source 2 and the connecting rod 51 can also be changed, and the adjustment of the light source 2 in multiple directions can be realized through the above arrangement.
  • the ring light source 2 is only one of the embodiments. According to the lighting requirements of different imaging methods, different lighting methods as mentioned above can also be used. Correspondingly, different installation forms and structures need to be changed , which will not be repeated here.
  • the beam splitting device can adopt a stereo beam splitter 4 or a beam splitter or a dichroic mirror. According to the requirements of different optical parameters, three types can be replaced and selected. In this embodiment, a stereo beam splitter 4 is used. And by changing the number of beam splitters, the number of imaging light paths can be changed. If it is necessary to adjust the three optical parameters of the same observation target 3, two stereoscopic beam splitters 4 arranged along the optical axis can be provided, so that three optical paths can be formed, and three imaging units 7 can be provided correspondingly.
  • a camera, a telecentric lens and a lens holder are used in this embodiment, and the telecentric lens is placed on the lens holder.
  • the observation part 11 of the observation container 1, the stereoscopic beam splitter 4 and the imaging unit 7 are all installed along the axis of the optical path.
  • an adjustment device which includes at least one of the following structures:
  • a position parameter adjustment structure arranged at the imaging unit 7 and/or the beam splitting device, to adjust the distance between each imaging unit 7 and the beam splitting device;
  • An optical element adjustment structure arranged between the imaging device and the beam splitting device and/or between the illuminator and the sample observation device, to adjust the polarization state and/or wavelength band and/or brightness of the light beam;
  • An imaging unit adjustment structure the imaging unit 7 includes a camera and a lens/multiple different lenses, and the imaging unit adjustment mechanism is used to switch the positions of multiple lenses in an imaging unit 7;
  • the beam splitting device adjustment structure the beam splitting device is one or more three-dimensional beam splitters 4 or beam splitters or dichroic mirrors, the beam splitting device adjusting structure is used to adjust the number or type of the beam splitting device.
  • the adjusting device includes a position parameter adjusting structure, an optical element adjusting structure, an imaging unit adjusting structure and a beam splitting device adjusting structure.
  • the position parameter adjustment structure includes an adjustment track arranged along the optical axis direction of the outgoing beam of the beam splitting device, and the imaging unit 7 that needs to adjust the distance position is slidably fitted on the adjustment track to adjust the distance with the beam splitter. distance between devices.
  • the optical element adjustment structure includes a mounting part, and the mounting part is detachably connected/replaceably connected with one or more polarizers, attenuation sheets, and optical filters, and the polarizers, attenuation sheets, and optical filters are all sequentially installed on each imaging On the optical path between unit 7 and the fractionator.
  • the installation part includes a mounting frame, and a vertical mounting plate is connected to the mounting frame in rotation, and each mounting plate is detachably fixed with a plurality of polarizers or attenuation plates or filters with different parameters or quantities. You can switch between different optical parameters of the same optical path by installing the disk.
  • the imaging unit adjustment structure is similar to the installation part structure of the optical element adjustment structure. Multiple cameras with different magnifications or gains or exposure times are installed on a rotatable mounting disk. By rotating the mounting disk, the same imaging unit can be realized. Switching of cameras with different parameters in .
  • the adjustment of the beam splitter is mainly realized by the cylinder, that is, the adjustment structure of the beam splitter includes a plurality of cylinders, and the end of the telescopic rod of each cylinder is connected with a mounting plate, and a three-dimensional beam splitter 4 or a beam splitter or beam splitter is positioned on the mounting plate.
  • Dichroic mirrors when the beam splitter needs to be adjusted, because the beam splitters driven by the ends of multiple cylinders are different, thus driving different cylinders, they can be combined to form different numbers or types of beam splitters, such as in a certain Only a single stereoscopic beam splitter 4 is placed in one optical path, or two stereoscopic beam splitters 4 can be placed in sequence, or the stereoscopic beam splitter 4 can be replaced with a dichroic mirror.
  • Embodiment 2 the device for constructing image pairs or image clusters or video pairs, differs from Embodiment 1 in that Embodiment 1 is a general-purpose device that can satisfy most optical parameter adjustments and has wide applicability, but the structure is also It is relatively complicated, and part of the structure is simplified in this embodiment, which is a customized type formulated for special observation situations.
  • the observation of plankton is taken as an example, and the purpose is to obtain image pairs or image clusters or video pairs of plankton in the sample solution.
  • the device comprises an illuminator, an observation container 1, a beam splitter, an imaging device and an adjustment device, the structure of the illuminator and the observation container 1 is the same as in Embodiment 1, and the beam splitter is a three-dimensional beam splitter 4 ,
  • the imaging device contains two groups of telecentric lenses, a camera and a lens holder.
  • the adjustment device (not shown in the figure) includes a position parameter adjustment structure and an imaging unit adjustment structure.
  • the position parameter adjustment structure is an adjustment track laid under each imaging device.
  • the imaging unit adjustment structure includes a fixed column 5, and the column 5 A fixed disk is connected with the upper rotation, and multiple lenses with different magnifications are fixedly mounted on the fixed disk.
  • the position of the two-way lens and one of the cameras can be adjusted by adjusting the track, and the distance between it and the beam splitter can be changed to obtain paired images with different degrees of defocus;
  • the rotation of the disk switches the lenses of different magnifications, so as to obtain paired images of different magnifications. Both can provide original images for subsequent image alignment processing.
  • two three-dimensional beam splitters 4 placed in sequence may also be provided, and the splitting ratio of the first-stage beam splitter is 1:2, and the splitting ratio of the second-stage beam splitter is 1: 1.
  • the optical signal emitted from the observation target 3 is divided into three paths, and the corresponding imaging device also includes three groups of imaging units 7 .
  • the adjustment device includes a position parameter adjustment structure and an imaging unit adjustment structure.
  • the position parameter adjustment structure is an adjustment track laid under each imaging device;
  • the imaging unit adjustment structure includes a fixed column 5, and a fixed plate is rotatably connected to the column 5 A plurality of lenses with different magnifications are fixedly mounted on the fixed plate, and different magnifications are realized by rotating the fixed plate.
  • three images can be obtained in one imaging, and the three images can form two pairs of images, which is more convenient.
  • the three optical paths can also be used to adjust the magnification or the distance at the same time, so that three clustered images with different magnifications or three clustered images with different defocus degrees can be obtained.
  • the optical parameters to be adjusted are also different, so the type and quantity of the beam splitting device and the number of imaging units 7 can be adjusted according to actual requirements, and the adjustment can be changed accordingly The composition and installation location of the device.
  • Embodiment 3 image construction method, it adopts the device in embodiment 1 or embodiment 2, concrete steps are:
  • Step 1 put the solution of the observation sample into the observation container 1, install and fix the illuminator and turn it on, the information of the observation sample is divided into multiple beams of signal light by the beam splitter and emitted, and each beam of light emitted by the beam splitter passes through different
  • the imaging unit 7 forms an object image containing said information.
  • the optical parameter is at least one of optical resolution, defocus distance, brightness, polarization state, color, spectrum, and fluorescence.
  • Step 2 changing the optical parameters of the different light beams emitted by the beam splitting device when imaging separately, so that different imaging units 7 simultaneously form target images with different optical parameters. Observe paired or clustered images of the sample.
  • the present embodiment provides the following control mode:
  • the camera working in the free running mode outputs the exposure status signal, triggers the non-emitting camera and illuminator working in the external trigger mode, and realizes the simultaneous acquisition of image pairs of stationary or moving targets;
  • the cameras of the multi-channel imaging system work in the external trigger mode at the same time, it can be set to generate a trigger signal through an external signal source, trigger two cameras and illuminator to work at the same time, and collect image pairs.
  • the external signal source can be a shutter button triggered manually, or an illuminator start signal automatically recognized.
  • optical parameters Regardless of whether it is two optical paths or multiple optical paths, at least the following methods can be used when adjusting optical parameters:
  • At least one pair of imaging units 7 use lenses with the same magnification, one of which uses a color camera and the other uses a monochrome camera, adjust the two imaging units
  • the focal planes of unit 7 are coincident and imaged at the same time;
  • At least one pair of imaging units 7 should use lenses with the same magnification, and one or more optical filters should be inserted between one of the imaging units 7 and the beam splitting device , and select light beams of different wavelength bands to enter the imaging unit 7 for imaging, and then adjust the two imaging units 7 to image clearly and simultaneously image;
  • At least one pair of imaging units 7 should use lenses with the same magnification, adjust the exposure time and gain of one of the imaging unit 7 cameras, and then adjust the imaging of both imaging units 7 to be clear After that, imaging at the same time;
  • At least one pair of imaging units 7 should use lenses with different magnifications, and then adjust the focal planes of the two imaging units 7 to overlap and adjust the depth of field to within the illumination range of the observed sample , while imaging;
  • At least one pair of imaging units 7 adopts lenses with the same magnification, and the focal planes of the two imaging units 7 are adjusted to coincide, and one of the imaging units 7 is fixed.
  • Another imaging unit 7 moves along the optical axis direction of the light beam, and the two imaging units 7 image simultaneously at different positions;
  • At least one pair of imaging units 7 should use lenses with the same magnification, and adjust the light beam illuminating the observation sample to monochromatic light, and replace the beam splitting device with two
  • the dichroic mirror is used to adjust the imaging of the two imaging units 7 to form images at the same time;
  • the device is a three-way or more imaging unit 7, the above-mentioned multiple optical parameters can be adjusted for one observation target 3 at the same time, and multiple images can be obtained at one time to form a clustered image; What is more important is that two imaging units 7 perform imaging, then only one optical parameter can be adjusted at a time to obtain a pair of images.
  • Step 3 Align multiple images of the same observation target 3 to obtain an image pair or image cluster or video pair for each observation target 3 . Due to errors in the assembly and processing of parts of the multi-channel imaging part, and the inconsistency of the magnification of the imaging part, the image of the same target in the image pair or image cluster or video pair collected by the multi-channel imaging part is not complete in the image space.
  • the specific method of alignment processing is as follows:
  • step 2 perform basic preprocessing on the paired images or clustered images obtained by camera imaging in step 2, such as color correction, background subtraction, CLAHE contrast enhancement, etc.;
  • the target detection algorithm commonly used in the prior art to detect the observation target 3 on one image in the paired images or clustered images, and obtain the target regions (ROIs) of multiple observation targets 3 , wherein, if the paired images or the clustered images If the cluster images contain images with different magnifications, images with high magnifications should be selected for target detection;
  • the sharpness judgment of the image detected by the observation target 3 can be judged manually, or can be judged and screened by means of an algorithm, and the clear target area of the judgment result is retained, and the blurred target area of the judgment result is discarded. If there is no clear target area, go back to the previous step and select an image again;
  • the coordinates of the four vertices of the target area in the original image are extracted during the observation target detection, and then the coordinates of the four vertices are transformed using the registration model to obtain paired images or clusters Four vertex coordinates of the same target area on the remaining images in the image;
  • the imaging coordinates of an observation target 3 in all images are known, and can be cut and saved according to the coordinates, and the image pair or image cluster or video pair of the observation target 3 can be obtained.
  • a registration model is needed.
  • the construction of the registration model mainly relies on the target 6 to assist acquisition.
  • the target 6 provides multiple planes/spaces that can be detected or identified feature, and assign coordinate information to each of the features.
  • the size of the target 6 is based on the principle that it can completely cover the field of view of the imaging unit 7 after being placed into the observation container 1.
  • the distribution and quantity of the features are based on the resolution and the viewing angle Depends on the size of the field.
  • the target 6 can adopt different forms and structures, such as the dark-field imaging illumination in Embodiment 2.
  • the target 6 is a three-dimensional structure, which includes a vertical flat plate, and a plurality of protrusions are arranged on the flat plate. Uniformly distributed quadrangular prism, each corner point of the prism can be detected or identified; if it is in the form of bright field illumination, the target 6 can be a two-dimensional grid, and each intersection point of the grid can be detected or identify.
  • multiple features distributed according to a certain spatial law can be constructed in the field of view, and registration can be established through a series of standard processes of image registration such as feature extraction, feature matching, and least squares solution.
  • the model can obtain the coordinate information of the same observation target 3 in the rest of the images after obtaining the coordinate information of the four points of the target area.
  • another image alignment processing method is also provided, and its specific processing method is:
  • step 2 perform basic preprocessing on the paired images or clustered images obtained by camera imaging in step 2, such as color correction, background subtraction, CLAHE contrast enhancement, etc.;
  • paired images or clustered images with the same magnification select one of the images for target detection and crop the target region (ROI).
  • ROI target detection and crop the target region
  • the sharpness judgment of the image detected by the observation target 3 can be judged manually, or can be judged and screened by means of an algorithm, and the clear target area of the judgment result is retained, and the blurred target area of the judgment result is discarded. If there is no clear target area, go back to the previous step and select an image again;
  • the clear target area is used as a template to perform template matching with other images in the paired image or clustered images, and the highest matching area in the image is cut out, and
  • the image of the target area in the previous step constitutes an image pair or image cluster or video pair and saves it;
  • Embodiment 4 image construction method, it adopts the device in embodiment 2, and the difference with embodiment 3 is that two three-dimensional beam splitters 4 placed in sequence are provided, and the light splitting ratio of the first-order beam splitter is 1:2, the light splitting ratio of the second-stage beam splitter is 1:1, so that the optical signal emitted from the observation target 3 is divided into three paths, and the corresponding imaging device also includes three groups of imaging units 7 .
  • control method is as follows:
  • the imaging unit adjustment structure is used to change the lens of the three-way imaging unit 7 to form three different magnifications, and three clusters of the same observation target 3 with different magnifications can be obtained after simultaneous imaging image;
  • the three-way imaging part uses telecentric lenses with the same magnification, and inserts band-pass filters of different bands into two of them through the installation part in the optical element adjustment structure, and the remaining one does not Without adding optical components, the camera collects images at a fixed frame rate, and obtains clustered images of the same target containing different bands;
  • the three-way imaging system uses telecentric lenses with the same magnification, and both the first and second beam splitters use dichroic mirrors, but the two dichroic mirrors Reflection and transmission wavelengths are different, the camera captures images at a certain frame rate, and imaging at the same time can obtain clustered images of the same target containing fluorescence and scattered light of different bands;
  • the three-way imaging part uses a telecentric lens with the same magnification, uses polarized light for illumination, and adjusts the position of the installation part in the three-way imaging part through optical elements.
  • Add polarizers to two paths adjust the angles of polarizers in the imaging part of the two paths, but the angles of the polarizers between the two paths are different, and collect images at the same time, you can get the same target including non-polarized images and clusters with different polarization angles image.
  • optical parameters for three-way imaging, different optical parameters can also be combined, such as:
  • Two of the three-way imaging systems use telecentric lenses of the same magnification, and the last one uses telecentric lenses of different magnifications for imaging.
  • the three-way imaging part adopts a telecentric lens with the same magnification.
  • a band-pass filter is inserted in one of the paths, and a polarizer is inserted in the other path.
  • the remaining path does not include optical elements.
  • the camera Acquire images at a fixed frame rate, and obtain clustered images of the same target including different wave bands and different polarization states.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Microscoopes, Condenser (AREA)
  • Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)

Abstract

Un procédé de construction d'image, comprenant les étapes consistant à : obtenir une image cible ; changer des paramètres optiques de différents faisceaux lumineux émis par un appareil de division de faisceau lorsque chaque faisceau lumineux est imagé, de telle sorte que différentes unités d'imagerie (7) forment simultanément des images cibles avec différents paramètres optiques, et des images appariées ou des images regroupées du même échantillon observé sont obtenues ; et effectuer un traitement d'alignement sur une pluralité d'images de la même cible observée (3), de façon à obtenir une paire d'images ou une grappe d'images ou une paire vidéo de chaque cible observée (3). Au moyen de la configuration ci-dessus, une pluralité d'images appariées ou d'images regroupées présentant des exigences différentes peut être formée au moyen de l'ajustement de différents paramètres optiques ou de différentes valeurs numériques du même paramètre optique, et un traitement d'enregistrement d'image est ensuite effectué pour obtenir un effet d'alignement, de sorte que les exigences de diversité, de haute précision et de quantité d'apprentissage profond sont satisfaites, et un schéma de base faisable pour construire un ensemble de données est proposé.
PCT/CN2021/138107 2021-09-26 2021-12-14 Procédé de construction d'image WO2023045116A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111131952.3A CN115880207A (zh) 2021-09-26 2021-09-26 图像构建方法
CN202111131952.3 2021-09-26

Publications (1)

Publication Number Publication Date
WO2023045116A1 true WO2023045116A1 (fr) 2023-03-30

Family

ID=85719266

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/138107 WO2023045116A1 (fr) 2021-09-26 2021-12-14 Procédé de construction d'image

Country Status (2)

Country Link
CN (1) CN115880207A (fr)
WO (1) WO2023045116A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103959040A (zh) * 2011-08-21 2014-07-30 摩巴尔欧西提有限责任公司 在智能手机上附接光学相干断层成像系统
CN107340546A (zh) * 2017-07-24 2017-11-10 南京信息工程大学 一种水下探测分孔径双ccd实时偏振成像装置及方法
CN207529042U (zh) * 2017-12-15 2018-06-22 中山联合光电科技股份有限公司 一种不同方向光路同时成像的光学镜头
US20190223728A1 (en) * 2018-01-19 2019-07-25 The Regents Of The University Of California Optical coherence tomography for cancer screening and triage
CN110175971A (zh) * 2019-05-27 2019-08-27 大连海事大学 一种多光谱单像素成像的深度学习图像重构方法
CN110927945A (zh) * 2019-11-28 2020-03-27 清华大学 三维宽视场和高分辨层析成像方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103959040A (zh) * 2011-08-21 2014-07-30 摩巴尔欧西提有限责任公司 在智能手机上附接光学相干断层成像系统
CN107340546A (zh) * 2017-07-24 2017-11-10 南京信息工程大学 一种水下探测分孔径双ccd实时偏振成像装置及方法
CN207529042U (zh) * 2017-12-15 2018-06-22 中山联合光电科技股份有限公司 一种不同方向光路同时成像的光学镜头
US20190223728A1 (en) * 2018-01-19 2019-07-25 The Regents Of The University Of California Optical coherence tomography for cancer screening and triage
CN110175971A (zh) * 2019-05-27 2019-08-27 大连海事大学 一种多光谱单像素成像的深度学习图像重构方法
CN110927945A (zh) * 2019-11-28 2020-03-27 清华大学 三维宽视场和高分辨层析成像方法及装置

Also Published As

Publication number Publication date
CN115880207A (zh) 2023-03-31

Similar Documents

Publication Publication Date Title
CN106896489B (zh) 基于波长复用的频域拼贴显微系统及其方法
US8154718B2 (en) Apparatus and method for inspecting micro-structured devices on a semiconductor substrate
JP5599314B2 (ja) 試料の検査のための方法および光学装置
JP2020509403A5 (fr)
US9013570B2 (en) Microscope and area determination method
US20150185460A1 (en) Image forming method and image forming apparatus
JP6972188B2 (ja) 異なるサイズのスライド用の調整可能なスライドステージ
JP2021193459A (ja) 二重光学経路および単一撮像センサを使用した低解像度スライド撮像、スライドラベル撮像および高解像度スライド撮像
JP7004808B2 (ja) スライドガラスの走査および処理のための対向縁部システム
JP6940696B2 (ja) 二次元および三次元の固定式ｚ走査
US11740447B2 (en) Illumination display as illumination source for microscopy
KR20200041983A (ko) 실시간 오토포커스 포커싱 알고리즘
CN109407297B (zh) 一种基于可编程led阵列照明的多模式光场显微成像方法
CN101627899A (zh) 微循环双通道仿立体伪彩色显微视频成像系统
JP2002267940A (ja) 倒立型顕微鏡システム
US8508589B2 (en) Imaging systems and associated methods thereof
US20180139366A1 (en) System and method for light sheet microscope and clearing for tracing
JP6952891B2 (ja) ２×３および１×３スライド用のカルーセル
US11356593B2 (en) Methods and systems for single frame autofocusing based on color- multiplexed illumination
WO2023045116A1 (fr) Procédé de construction d'image
CN105308492B (zh) 光观察装置、用于其的摄像装置以及光观察方法
CN112731641A (zh) 多模式成像手机显微镜装置
JP2004054108A (ja) 光路分割光学素子とこれを用いた顕微鏡
CN1474210A (zh) 激光微作用数字显微装置
Liao Imaging Innovations for Whole-Slide and Hyperspectral Microscopy

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21958227

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE