CN108873302B - Automatic correction and refocusing method of light field microscopic image - Google Patents
Automatic correction and refocusing method of light field microscopic image Download PDFInfo
- Publication number
- CN108873302B CN108873302B CN201810691605.8A CN201810691605A CN108873302B CN 108873302 B CN108873302 B CN 108873302B CN 201810691605 A CN201810691605 A CN 201810691605A CN 108873302 B CN108873302 B CN 108873302B
- Authority
- CN
- China
- Prior art keywords
- image
- images
- sub
- light field
- micro
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 title claims abstract description 15
- 238000001000 micrograph Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 2
- 230000007547 defect Effects 0.000 abstract description 11
- 238000005096 rolling process Methods 0.000 abstract description 4
- 238000003491 array Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000002950 deficient Effects 0.000 description 3
- 238000000386 microscopy Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 235000020280 flat white Nutrition 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
- G02B21/367—Control or image processing arrangements for digital or video microscopes providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Optics & Photonics (AREA)
- Theoretical Computer Science (AREA)
- Microscoopes, Condenser (AREA)
- Image Processing (AREA)
Abstract
本申请提供一种光场显微图像的自动校正和重新聚焦方法，(a)将捕获的光场图像通过自动校正算法自动分割成微图像；(b)把步骤(a)中的微图像分割成子图像；(c)通过滚动和添加算法将步骤(b)中的子图像重新聚焦生成重新聚焦后的图像；通过使用Ostu算法将彩色的光场图像转换为二值图像，不仅用于检测光场图像的缺陷，且很容易计算每个微透镜的中心和微图像的半径，节省时间并提高校正精度；通过滚动和添加算法将更多噪点添加到离焦区域，从而在对焦区域和离焦区域之间提供更多对比。
The present application provides an automatic correction and refocusing method for a light field microscopic image, which includes: (a) automatically segmenting a captured light field image into micro images by an automatic correction algorithm; (b) segmenting the micro image in step (a) into sub-images; (c) refocusing the sub-image in step (b) by a rolling and adding algorithm to generate a refocused image; by using the Ostu algorithm to convert a color light field image into a binary image, it is not only used to detect defects in the light field image, but also easy to calculate the center of each microlens and the radius of the micro image, saving time and improving correction accuracy; by adding more noise points to the defocused area by the rolling and adding algorithm, more contrast is provided between the in-focus area and the defocused area.
Description
Technical Field
The invention relates to the technical field of light field microscopic images, in particular to a method for automatically correcting and refocusing an image.
Background
Conventional microscopes have some limitations, when using high power lenses or larger aperture lenses, the depth of field is very shallow, requiring an adjustable platform to move objects up and down to view different layers of objects, resulting in the organisms or photosensitive samples being difficult to observe or capture, and conventional optical microscopes have only one directional projection, and cannot provide 3D information.
Currently, light field microscopy imaging systems have been developed that capture information of light intensity, color, and depth using microlens arrays. To analyze light field information within images captured by a light field microscopy imaging system, an image processing method should be performed to calculate depth information and refocus the light field microscopy images; in the prior art, the micro lens is generally corrected first. The traditional correction method is to manually input the center and the radius of the micro lens, and because the micro lens array has hundreds of thousands of micro lenses, the manual input is time-consuming and inaccurate; after correcting the microlenses, the shift-add algorithm is used to calculate the depth information of the light field image, generating a refocused image, defects in the microlens array are ignored, they should be considered as noise, but generally they are not detected and still used to calculate the depth information.
A light field microscopic imaging system is composed of five basic components: microlens arrays, special microlenses, digital cameras, optical microscopes and LED illumination. The microlens array is mounted in front of the CMOS of the camera, fig. 1 is an image at F number of different microlens arrays in a light field microscope, fig. 1(a) a schematic of a light field microscope; FIG. 1(b) when the F number of the microlens array is lower than the F number of the image side of the objective lens, the image gap is relatively large; FIG. 1(c) when the F number of the microlens array is higher than the F number on the image side of the objective lens, the images overlap; fig. 1(d) is continuous between images when the F-number of the microlens array matches the image-side F-number of the objective lens. For the cases in fig. 1(b) and 1(d), if the center and radius of the microimage can be found, the microimage can be segmented; however, in the case of fig. 1(c), since the microimages overlap, it is impossible to correctly divide the microimages.
Disclosure of Invention
The invention provides an automatic correction and refocusing method of a light field microscopic image, which can detect image defects, automatically, quickly and accurately correct the image defects and refocus by using a superposition algorithm of the invention.
The invention relates to an automatic correction and refocusing method of a light field microscopic image, which comprises the following steps:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
Preferably, the automatic correction algorithm in step (a) is implemented by locating the center of the microimage and calculating the radius of the microimage.
Preferably, the step (a) includes converting the colored light field image into a binary image using Otsu's algorithm for locating the center of the microimage and calculating the radius of the microimage.
Preferably, the automatic correction algorithm in step (a) is used to detect defects in the image of the optical field.
Preferably, in step (b), the sub-image is a specific view of the light field image.
Preferably, in the step (b), the microimages are divided into sub-images according to the centers of the microimages and the radii of the microimages, and the relationship between the sub-images and the microimages is defined as follows:
Si，j(x，y)＝Mx，y(i，j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
Preferably, in the step (c), the number of the sub-images is n, and n is a positive integer.
Preferably, in step (c), the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively.
Preferably, the addition of noise outside the focus area is achieved by the refocused image calculated using a rolling addition algorithm.
Preferably, an image processing module comprises the automatic correction and refocusing method of light field microscopy images of the present application.
The beneficial effect of adopting above-mentioned technical scheme is:
according to the method, the color light field image is converted into the binary image by using the Otsu algorithm, the center of the microimage is positioned, the radius of the microimage is calculated, the defect of the light field image is detected, the center of the black hole is used, the center of each micro lens and the radius of the microimage are easily calculated, time is saved, correction precision is improved, the defect in the light field image is automatically detected, and more accurate depth information calculation is provided; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic of an image at F-numbers of different microlens arrays in a light field microscope;
FIG. 2 is a flow chart of a method of automatic correction and refocusing of a light field microscopic image in accordance with the present invention;
FIG. 3 is a white background image and partial view captured by a microlens array;
FIG. 4 is a schematic diagram of the binary image of FIG. 3 after Otsu's algorithm;
FIG. 5 is a schematic diagram of the generation of a sub-image;
FIG. 6 is a schematic illustration of a sub-image being re-focused by a rolling addition algorithm;
FIG. 7 is a schematic illustration of a light field microscopic image;
FIG. 8 is a schematic diagram of the detection results of black holes and defects in a light field microscopic image;
FIG. 9 is a schematic diagram of a microimage acquired after auto-calibration;
FIG. 10 is a schematic illustration of the generation of a sub-image after the microimage generation of FIG. 9;
FIG. 11 is a schematic illustration of a refocused image generated by the sub-image of FIG. 10.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
A method for automatic correction and refocusing of light field microscopy images, as shown in fig. 2, comprising the steps of:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (a), as shown in fig. 3a, a flat white paper is photographed using a light field microscope, wherein defects and black holes are present in the image, and some microlenses are defective in the upper left area of fig. 3a and should not be used to calculate depth information; the black holes in FIG. 3b show the non-overlapping areas of the microimages.
To find the center of the microimage, the black hole needs to be detected first. As shown in fig. 4, by converting the color image into a binary image using Otsu's algorithm, it is possible to highlight the black hole and the defective region, find the outline of each black region, and calculate the number of pixels of each region. The number of pixels of the black hole should be smaller than that of the defect, and therefore, all the regions having similar and relatively smaller number of pixels are located and are detected black holes, and using the center of the black hole, it is easy to calculate the center of each microlens and the radius of the microimage.
For step (b), the sub-image is a particular view of the light field image; dividing the micro image into sub images according to the center of the micro image and the radius of the micro image, wherein the relation between the sub images and the micro image is defined as follows:
Si，j(x，y)＝Mx，y(i，j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
As shown in fig. 5, it is shown how a sub-image S can be obtained from a micro-image calculated from a light field image consisting of 3 columns and 2 rows of micro-images, i.e. a light field image consisting of six micro-images, each taking the pixels of (-1, -1)-1,-1
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; if the image acquired after the sub-image S is moved in the x direction is defined as S', and the image acquired after the sub-image S is moved in the xy direction is defined as S ″, it can be derived:
S″＝RollXY(S，i，j)＝RollY(S′，j)
S′＝RollX(S，i)
fig. 6 shows how the rolxy (S, 2, -1) is calculated from the sub-image S; moving 2 pixels in the x-direction yields S ', and moving S' 1 element in the y-direction yields the sub-image S ". The roll and add algorithm, which is different from the shift and add method, fills the pixels that move out of the region with empty pixels instead of with a value of 0, by which noise is added to the incoherent pixels, making the objects on the focal plane clearer and the objects on the non-focal plane more blurred, thereby providing more contrast between the in-focus and out-of-focus regions.
Example 2
In order to further verify the technical effect of the scheme, further experiments are carried out on an automatic correction and refocusing method of a light field microscopic image.
Automatically segmenting a captured light field image into microimages through an automatic correction algorithm; as shown in fig. 7, a light field image captured by a light field microscopy imaging system; fig. 9 is a micro image segmented by an auto-correction algorithm.
Step (b) segmenting the microimage of step (a) into sub-images; by converting the color image into a binary image using Otsu's algorithm, as shown in fig. 8, it is possible to highlight black holes and defective regions, find the outline of each black region, and calculate the number of pixels of each region; FIG. 8 shows the detection result of black holes and the defects of the light field image; the black dots are detected black holes, and the result shows that the detection is very accurate, the detection speed is very high, and 5184 multiplied by 3456 resolution images with 120 ten thousand micro images can be detected in 2 seconds. The center of each microlens and the radius of the microimage are easily calculated by using the center of the black hole; the sub-images are segmented according to the center of the micro-image and the radius of the micro-image as shown in fig. 10.
Step (c) refocuses the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, RollX and RollY is let sub-picture Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; as shown in fig. 11, a refocused image calculated by the scroll and add algorithm is displayed; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (2)
1. A method for automatically correcting and refocusing a light field microscopic image, comprising the steps of:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm; the automatic correction algorithm is to convert the color image into a binary image by using an Otsu algorithm, find the outline of each black area, calculate the number of pixels of each black area, detect a black hole, and calculate the center of each microlens and the radius of the microimage by using the center of the black hole;
(b) segmenting the microimage of step (a) into sub-images;
the relationship of the sub-image to the microimage is defined as follows:
Si,j(x,y)＝Mx,y(i,j)
wherein S is a sub-image, M is a micro-image, i and j are pixel coordinates in the micro-image M, x is x columns of micro-images in the light field image, and y is y rows of micro-images in the light field image;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image;
the number of the sub-images is n, n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively;
the pixels that move out of the region are filled with empty pixels by a scroll and add algorithm by which noise is added to the incoherent pixels.
2. An image processing module comprising the method of automatic correction and refocusing of light field microscopy images of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810691605.8A CN108873302B (en) | 2018-06-28 | 2018-06-28 | Automatic correction and refocusing method of light field microscopic image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810691605.8A CN108873302B (en) | 2018-06-28 | 2018-06-28 | Automatic correction and refocusing method of light field microscopic image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108873302A CN108873302A (en) | 2018-11-23 |
| CN108873302B true CN108873302B (en) | 2021-02-05 |
Family
ID=64296789
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810691605.8A Active CN108873302B (en) | 2018-06-28 | 2018-06-28 | Automatic correction and refocusing method of light field microscopic image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108873302B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11030776B2 (en) * | 2019-02-01 | 2021-06-08 | Molecular Devices (Austria) GmbH | Calibration of a light-field imaging system |
| CN113219643B (en) * | 2021-05-11 | 2022-07-12 | 浙江大学 | Optical microscope focusing stabilization method and system based on incoherent imaging edge blurring |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013003826A1 (en) * | 2011-06-29 | 2013-01-03 | The Regents Of The University Of Michigan | Analysis of temporal changes in registered tomographic images |
| DE102011054106A1 (en) * | 2011-09-30 | 2013-04-04 | Jörg Piper | Method for producing variable phase contrast and/or dark-field imaging, using microscope, involves enabling light rings to make the lights pass through object, so that the summation image is arranged in the plane of intermediate image |
| CN105683815A (en) * | 2013-09-04 | 2016-06-15 | 卢门科有限责任公司 | Servo system, and encoder |
| CN104732563B (en) * | 2015-04-01 | 2017-06-27 | 河南理工大学 | A centerline detection method of rings based on double distance transformation and feature distribution in images |
-
2018
- 2018-06-28 CN CN201810691605.8A patent/CN108873302B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013003826A1 (en) * | 2011-06-29 | 2013-01-03 | The Regents Of The University Of Michigan | Analysis of temporal changes in registered tomographic images |
| DE102011054106A1 (en) * | 2011-09-30 | 2013-04-04 | Jörg Piper | Method for producing variable phase contrast and/or dark-field imaging, using microscope, involves enabling light rings to make the lights pass through object, so that the summation image is arranged in the plane of intermediate image |
| CN105683815A (en) * | 2013-09-04 | 2016-06-15 | 卢门科有限责任公司 | Servo system, and encoder |
| CN104732563B (en) * | 2015-04-01 | 2017-06-27 | 河南理工大学 | A centerline detection method of rings based on double distance transformation and feature distribution in images |
Non-Patent Citations (2)
| Title |
|---|
| Near-field scanning optical microscopy（MSOM）;E.Betziq,et al.;《Scanning Microscopy Technologies and Applications》;19881231;全文 * |
| 基于光场成像的三维测量方法的研究;王宇等;《仪器仪表学报》;20160615;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108873302A (en) | 2018-11-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5446797B2 (en) | Imaging device | |
| JP5173954B2 (en) | Image processing apparatus and image processing method | |
| US9088729B2 (en) | Imaging apparatus and method of controlling same | |
| JP5552214B2 (en) | Focus detection device | |
| RU2585235C2 (en) | Image capture devices | |
| US8730374B2 (en) | Focus detection apparatus | |
| CN107465866B (en) | Image processing apparatus and method, imaging apparatus, and computer-readable storage medium | |
| CN104041006B (en) | Image generating method and image forming apparatus | |
| US10321044B2 (en) | Image pickup apparatus and image pickup system with point image intensity distribution calculation | |
| JP2016038414A (en) | Focus detection device, control method thereof, and imaging apparatus | |
| JP6381266B2 (en) | IMAGING DEVICE, CONTROL DEVICE, CONTROL METHOD, PROGRAM, AND STORAGE MEDIUM | |
| CN107960120B (en) | Image processing apparatus, image capturing apparatus, image processing method, and storage medium | |
| CN107431755B (en) | Image processing apparatus, image capturing apparatus, image processing method, and storage medium | |
| CN108873302B (en) | Automatic correction and refocusing method of light field microscopic image | |
| JP6700986B2 (en) | Image processing device, imaging device, image processing method, and program | |
| CN107249089A (en) | Filming apparatus and method, video camera and logistics capturing system | |
| JP2015194736A (en) | Imaging apparatus and control method thereof | |
| CN104184936A (en) | Image focusing processing method and system based on light field camera | |
| TWI604221B (en) | Method for measuring depth of field and image pickup device using the same | |
| JP2015210285A (en) | Imaging device, manufacturing method of the same, program thereof and recording medium | |
| KR100764436B1 (en) | How to compare the sharpness of each color channel of an image for autofocusing | |
| CN113923445B (en) | Light field camera calibration method and system under shift imaging condition | |
| CN112866547B (en) | Focusing method and device, electronic equipment and computer readable storage medium | |
| JP5018932B2 (en) | Imaging device | |
| US10200622B2 (en) | Image processing method, image processing apparatus, and imaging apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |