Automatic correction and refocusing method of light field microscopic image
Technical Field
The invention relates to the technical field of light field microscopic images, in particular to a method for automatically correcting and refocusing an image.
Background
Conventional microscopes have some limitations, when using high power lenses or larger aperture lenses, the depth of field is very shallow, requiring an adjustable platform to move objects up and down to view different layers of objects, resulting in the organisms or photosensitive samples being difficult to observe or capture, and conventional optical microscopes have only one directional projection, and cannot provide 3D information.
Currently, light field microscopy imaging systems have been developed that capture information of light intensity, color, and depth using microlens arrays. To analyze light field information within images captured by a light field microscopy imaging system, an image processing method should be performed to calculate depth information and refocus the light field microscopy images; in the prior art, the micro lens is generally corrected first. The traditional correction method is to manually input the center and the radius of the micro lens, and because the micro lens array has hundreds of thousands of micro lenses, the manual input is time-consuming and inaccurate; after correcting the microlenses, the shift-add algorithm is used to calculate the depth information of the light field image, generating a refocused image, defects in the microlens array are ignored, they should be considered as noise, but generally they are not detected and still used to calculate the depth information.
A light field microscopic imaging system is composed of five basic components: microlens arrays, special microlenses, digital cameras, optical microscopes and LED illumination. The microlens array is mounted in front of the CMOS of the camera, fig. 1 is an image at F number of different microlens arrays in a light field microscope, fig. 1(a) a schematic of a light field microscope; FIG. 1(b) when the F number of the microlens array is lower than the F number of the image side of the objective lens, the image gap is relatively large; FIG. 1(c) when the F number of the microlens array is higher than the F number on the image side of the objective lens, the images overlap; fig. 1(d) is continuous between images when the F-number of the microlens array matches the image-side F-number of the objective lens. For the cases in fig. 1(b) and 1(d), if the center and radius of the microimage can be found, the microimage can be segmented; however, in the case of fig. 1(c), since the microimages overlap, it is impossible to correctly divide the microimages.
Disclosure of Invention
The invention provides an automatic correction and refocusing method of a light field microscopic image, which can detect image defects, automatically, quickly and accurately correct the image defects and refocus by using a superposition algorithm of the invention.
The invention relates to an automatic correction and refocusing method of a light field microscopic image, which comprises the following steps:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
Preferably, the automatic correction algorithm in step (a) is implemented by locating the center of the microimage and calculating the radius of the microimage.
Preferably, the step (a) includes converting the colored light field image into a binary image using Otsu's algorithm for locating the center of the microimage and calculating the radius of the microimage.
Preferably, the automatic correction algorithm in step (a) is used to detect defects in the image of the optical field.
Preferably, in step (b), the sub-image is a specific view of the light field image.
Preferably, in the step (b), the microimages are divided into sub-images according to the centers of the microimages and the radii of the microimages, and the relationship between the sub-images and the microimages is defined as follows:
Si,j(x,y)=Mx,y(i,j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
Preferably, in the step (c), the number of the sub-images is n, and n is a positive integer.
Preferably, in step (c), the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively.
Preferably, the addition of noise outside the focus area is achieved by the refocused image calculated using a rolling addition algorithm.
Preferably, an image processing module comprises the automatic correction and refocusing method of light field microscopy images of the present application.
The beneficial effect of adopting above-mentioned technical scheme is:
according to the method, the color light field image is converted into the binary image by using the Otsu algorithm, the center of the microimage is positioned, the radius of the microimage is calculated, the defect of the light field image is detected, the center of the black hole is used, the center of each micro lens and the radius of the microimage are easily calculated, time is saved, correction precision is improved, the defect in the light field image is automatically detected, and more accurate depth information calculation is provided; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic of an image at F-numbers of different microlens arrays in a light field microscope;
FIG. 2 is a flow chart of a method of automatic correction and refocusing of a light field microscopic image in accordance with the present invention;
FIG. 3 is a white background image and partial view captured by a microlens array;
FIG. 4 is a schematic diagram of the binary image of FIG. 3 after Otsu's algorithm;
FIG. 5 is a schematic diagram of the generation of a sub-image;
FIG. 6 is a schematic illustration of a sub-image being re-focused by a rolling addition algorithm;
FIG. 7 is a schematic illustration of a light field microscopic image;
FIG. 8 is a schematic diagram of the detection results of black holes and defects in a light field microscopic image;
FIG. 9 is a schematic diagram of a microimage acquired after auto-calibration;
FIG. 10 is a schematic illustration of the generation of a sub-image after the microimage generation of FIG. 9;
FIG. 11 is a schematic illustration of a refocused image generated by the sub-image of FIG. 10.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
A method for automatic correction and refocusing of light field microscopy images, as shown in fig. 2, comprising the steps of:
(a) automatically segmenting the captured light field image into micro images through an automatic correction algorithm;
(b) segmenting the microimage of step (a) into sub-images;
(c) refocusing the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (a), as shown in fig. 3a, a flat white paper is photographed using a light field microscope, wherein defects and black holes are present in the image, and some microlenses are defective in the upper left area of fig. 3a and should not be used to calculate depth information; the black holes in FIG. 3b show the non-overlapping areas of the microimages.
To find the center of the microimage, the black hole needs to be detected first. As shown in fig. 4, by converting the color image into a binary image using Otsu's algorithm, it is possible to highlight the black hole and the defective region, find the outline of each black region, and calculate the number of pixels of each region. The number of pixels of the black hole should be smaller than that of the defect, and therefore, all the regions having similar and relatively smaller number of pixels are located and are detected black holes, and using the center of the black hole, it is easy to calculate the center of each microlens and the radius of the microimage.
For step (b), the sub-image is a particular view of the light field image; dividing the micro image into sub images according to the center of the micro image and the radius of the micro image, wherein the relation between the sub images and the micro image is defined as follows:
Si,j(x,y)=Mx,y(i,j)
where S is the sub-image, M is the microimage, i, j are the pixel coordinates in the microimage M, x is the presence of x columns of microimages in the light field image, and y is the presence of y rows of microimages in the light field image.
As shown in fig. 5, it is shown how a sub-image S can be obtained from a micro-image calculated from a light field image consisting of 3 columns and 2 rows of micro-images, i.e. a light field image consisting of six micro-images, each taking the pixels of (-1, -1)-1,-1
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, which are let sub-images Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; if the image acquired after the sub-image S is moved in the x direction is defined as S', and the image acquired after the sub-image S is moved in the xy direction is defined as S ″, it can be derived:
S″=RollXY(S,i,j)=RollY(S′,j)
S′=RollX(S,i)
fig. 6 shows how the rolxy (S, 2, -1) is calculated from the sub-image S; moving 2 pixels in the x-direction yields S ', and moving S' 1 element in the y-direction yields the sub-image S ". The roll and add algorithm, which is different from the shift and add method, fills the pixels that move out of the region with empty pixels instead of with a value of 0, by which noise is added to the incoherent pixels, making the objects on the focal plane clearer and the objects on the non-focal plane more blurred, thereby providing more contrast between the in-focus and out-of-focus regions.
Example 2
In order to further verify the technical effect of the scheme, further experiments are carried out on an automatic correction and refocusing method of a light field microscopic image.
Automatically segmenting a captured light field image into microimages through an automatic correction algorithm; as shown in fig. 7, a light field image captured by a light field microscopy imaging system; fig. 9 is a micro image segmented by an auto-correction algorithm.
Step (b) segmenting the microimage of step (a) into sub-images; by converting the color image into a binary image using Otsu's algorithm, as shown in fig. 8, it is possible to highlight black holes and defective regions, find the outline of each black region, and calculate the number of pixels of each region; FIG. 8 shows the detection result of black holes and the defects of the light field image; the black dots are detected black holes, and the result shows that the detection is very accurate, the detection speed is very high, and 5184 multiplied by 3456 resolution images with 120 ten thousand micro images can be detected in 2 seconds. The center of each microlens and the radius of the microimage are easily calculated by using the center of the black hole; the sub-images are segmented according to the center of the micro-image and the radius of the micro-image as shown in fig. 10.
Step (c) refocuses the sub-images in step (b) by a scrolling and adding algorithm to generate a refocused image.
For step (c), the number of the sub-images is n, where n is a positive integer, and the relationship between the sub-images and the refocused image is as follows:
wherein R is the refocused image, α is a parameter controlling the focal plane distance, n is the number of sub-images, n is a positive integer; RollXY (S)i,jα i, α j) is a combination of RollX and RollY, RollX and RollY is let sub-picture Si,jShifting α i pixels in the x direction and α j pixels in the y direction, respectively; as shown in fig. 11, a refocused image calculated by the scroll and add algorithm is displayed; more noise is added to the out-of-focus region by the scrolling and adding algorithm, providing more contrast between the in-focus and out-of-focus regions, making objects in the focal plane clearer, but objects in the non-focal plane more blurred, as noise is added to the incoherent pixels.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.