AU2020103713A4

AU2020103713A4 - Digital imaging methods and system for processing agar plate images for automated diagnostics

Info

Publication number: AU2020103713A4
Application number: AU2020103713A
Authority: AU
Inventors: V. Ajantha Devi; Noor Zaman Jhanjhi; Rajalakshmi Krishnamurthi; Adarsh Kumar; Anand Nayyar; Kshira Sagar Sahoo
Original assignee: Devi VAjantha
Current assignee: Devi VAjantha
Priority date: 2020-11-27
Filing date: 2020-11-27
Publication date: 2021-02-18
Anticipated expiration: 2028-11-27

Abstract

DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS ABSTRACT Today, the infection is diagnosed manually, examining the bacterial growth on agar plates. Nevertheless, the Classifiers are designed for automated diagnostics using agar plate images. Input images should be of good quality and consistent in terms of scale, placement, perspective and rotation for precise classification. The Present invention disclosed here is Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104). The invention disclosed here investigates, whether a combination of image processing method can be used to match an input image to a predefined reference model. The invention was implemented to identify the key points required to record the input picture of the reference model. The key points found in the picture were the identification of the agar plate, its compartments and its rotation. The drawings illustrates that the recording of an image with the proper key points was sufficient to match the image of agar plates to a reference model, despite all scale, position, perspective or rotation variations. However, the accuracy was dependent on understanding the characteristics of the agar plate. Ultimately, invention proposes a method of converting images of agar plates on the basis of a predefined model instead of a reference image with image recording. 1/3 DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS DRAWINGS 101 Identify and Mask the Agar Plate Identify Comnpartment Edgesin 2 2103 Identify Agar Plate Orientation Img eistration Figure 1: Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics. Colriaant Border Following Validation RANdom Sample $ ROIMasing Detecion peraionsConsensus Figure 2: Workflow to Identify and Mask the Agar Plate.

Description

1/3

DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS DRAWINGS

101

Identify and Mask the Agar Plate

Identify Comnpartment Edgesin

2103 2 Identify Agar Plate Orientation

Img eistration

Figure 1: Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics.

Colriaant Detecion Border Following Validation peraionsConsensus RANdom Sample $ ROIMasing

Figure 2: Workflow to Identify and Mask the Agar Plate.

AUSTRALIA Patents Act 1990

COMPLETE SPECIFICATION INNOVATION PATENT DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS

The following statement is a full description of this invention, including the best method of performing it known to me:

I DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS FIELD OF INVENTION

[0001] The present invention relates to the technical field of Medical Image Processing of Biomedical Engineering.

[0002] Particularly, the present invention is related to Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics of the broader field of Image Enhancement in Medical Image Processing of Biomedical Engineering.

[0003] More particularly, the present invention is relates to Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics in which the Image Processing method is used for converting the Agar Plate Images on the basis of a predefined model instead of a reference image with image recording for Automated Diagnostics.

BACKGROUND OF INVENTION

[0004] Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics disclosed here is the most useful in Medical Image Analysis for Automated Diagnostics of the disease or the infections in the blood by the Agar Plate Images without manual analysis. Today, the infection is diagnosed manually, examining the bacterial growth on agar plates. Nevertheless, the classifiers are designed for automated diagnostics using agar plate images. Input images should be of good quality and consistent in terms of scale, placement, perspective and rotation for precise classification.

[0005] Processing digital images is about pixel manipulation, Improving, analyzing, or in any way, extracting information from an image, for example.

[0006] In image processing, altering the colour of an image is an important task and often involves eliminating dominant or undesirable colours in a frame. In the pictures, objects which have different colours compared to reality. An algorithm called Simplest Color Balance (SCB) for colour balance works within a [0, 255] range by extending red, green, and blue values. It is believed that pixels with high values produce high levels of the colour in question and vice versa. The same definition can be used to match luminosity, where grayscale-values are used instead of RGB-values. For instance, an image with low brightness could not have any pixels close to 255. The brightness can be either increased or decreased by extending the scale [0, 255].

[0007] To extract particular data from objects in images, edges need to be differentiated. The real edges will be easier to distinguish by decreasing the number of false edges. If the aim is to find the most extreme edge, blur philtres and morphological operations could be used as an extension, as the edge of an agar plate. The algorithm of Hough Transform is designed to find simple shapes within an image and to extract the coordinates or an object's region. The operator of the Canny is based on a multi-stage algorithm that was initially designed to detect image edges.

[0007a] The first stage of Canny is to apply a Gaussian blur filter to reduce any image noise occurring, otherwise, pixel noise could be falsely detected as an edge. The canny algorithm uses a kernel, which is a convolution matrix of n pixels, to search all pixels in an image. Each pixel is evaluated, and its pixel value is replaced with a weighted mean of its neighbouring pixels. A two-dimensional Gaussian function is generally used to remove image noise as given by the Equation 1.

G(x, y)=exp[-(x 2+Y 2/2s 2]/2ps 2 Equation 1

[0007b] A Sobel kernel is used to identify all intensity gradients following the applied blur. Sobel looks for areas in the picture with its kernel that have a high gradient, i.e. a distinct and sudden change in colour (in RGB) or brightness. Figure 2 shows how both the x-axis and the y-axis function in the method. Each value within the kernel is multiplied by the value in the image at the respective location. The sum of the matrix gives the value of a gradient. A higher value means an edge that is more defined. For the determination of the angle of the tip, the derivative of the respective edges is then determined.

-J

[0007c] Although most edges are identified by the Sobel philtre, generally, only the most prominent are interesting. The production can have both thick and thin edges, and any false edges can be sorted using non-maximum suppression. The canny algorithm looks through the image, following the direction of the edge, by using the gradient intensity matrix determined in the previous step. The algorithm tests neighbouring pixels to the left and right of the edge for every pixel. If a neighbouring pixel is more intense than the current pixel in the same direction, the most intense pixel is retained. An example is shown in Figure 3 where pixels with intensity values of 170 and 115 would be dismissed and 255 would be retained.

[0008] Canny Edge Detection and related techniques to detect edges in photographs have been the subject of several recent innovations, conducted a analysis of various edge detection methods for various applications for image processing. Since edge detection methods are problem-oriented, the same edge detection algorithm cannot be implemented for all types of images. Since performing edge detection in noisy images is difficult, the authors compare different methods of edge detection with their benefits and limitations. Excellent results were generated by Canny edge detection, particularly under noisy conditions.

[0009] In addition, several recent innovations have concentrated on enhancing Canny Edge Detection to help distinguish edges in noisy images, indicates an enhancement by using a modified median philtre rather than Gaussian smoothing. The algorithm was able to successfully eliminate noise from an ultrasound image of a kidney with optimal threshold values for the canny operator. A method of adaptive threshold selection to retain more useful edges and more stable noise for the Canny Edge Detection algorithm is also invented previously.

[0010] In pupil recognition invention, suggests using a blend of the two Canny Edge Detection and Hough Transform techniques to detect pupils in pictures. The canny operator is used to define a pupil's tip, while the precise location is identified by Hough Transform. Under various lighting conditions, the proposed algorithm managed to match circles of different eye images accurately. In addition, an algorithm that, using Hough Transform and Canny Edge detection, can detect and outline the outer edge of the pupils in human eyes. The algorithm was tested on 100 human eyes and provided a successful result of 95 percent.

[0011] For model fitting, RANSAC (RANdom Sample Consensus) is a common method of choice for finding ellipses in images. In natural and noisy conditions and for variable lighting, the use of RANSAC offers high accuracy and low running time. An ellipse detection method using RANSAC for the detection of multiple ellipses in images that achieves high precision and computational cost. Within two stages, the algorithm works. Firstly, area segmentation and identification of contours are applied. Secondly, a modified RANSAC is applied to five randomly selected pixels to form an accurate ellipse with each contour segment found.

[0012] There are various methods of picture registration that are mostly used in medical imaging, automatic target identification, and computer vision. The Image registration methods are broken down into two categories, Area-based and Feature based. In images such as lines, points, and contours, feature-based techniques find correspondence in salient characteristics. However, area-based techniques emphasise function matching without the detection of salient objects.

[0013] The present invention disclosed here is for understanding the use of Image Processing method in automatic diagnostics of Agar Plate Images. This disclosure uses Image Processing methods such as Edge Detection, Shape Detection, and Image Registration.

[0014] Today, the infection is diagnosed manually, examining the bacterial growth on agar plates. Nevertheless, the Classifiers are designed for automated diagnostics using agar plate images. Input images should be of good quality and consistent in terms of scale, placement, perspective and rotation for precise classification.

[0015] Referring to Figure 1, The Present invention disclosed here is Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104). The invention disclosed here investigates, whether a combination of image processing method can be used to match an input image to a predefined reference model. The invention was implemented to identify the key points required to record the input picture of the reference model. The key points found in the picture were the identification of the agar plate, its compartments and its rotation.

SUMMARY OF INVENTION

[0016] The Present invention disclosed here is Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104) discloses and investigates, whether a combination of image processing method can be used to match an input image to a predefined reference model in Agar Plate Images for Automatic Diagnostics of the infection or disease present in the blood.

[0017] Referring to Figure 1, Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104) provides the fundamental image processing steps involved to process the Agar Plate Images for Automatic Diagnostics.

[0018] The first step used in the invention is Identify and Mask the Agar Plate (101) is to describe the Agar Plate and Provides an elliptical contour of the ROI defining key points.

[0018a] The first step used in the invention uses Canny operator to intensify the edges, and reduce any image noise, with contrast, brightness, and colour all balanced.

[0018b] Using RANSAC, the validated contour will then be fitted to an elliptical model, which should provide a better approximation of the external contour, providing the desired key-point range. Finally, a binary mask will be applied to the image with the ROI identified by the contour key-points from past steps, removing the unwanted context.

[0019] The Second step used in the invention is Identify Compartment Edges (102) is to identification of the edges of the compartment and the centre point on the Agar Plate. It is possible to segment each compartment divider as two straight lines, which are

U

compartment edges. Therefore, these shapes can be defined by using Hough Transform. To correctly identify the edges and minimize irrelevant noise that may occur, some pre processing might be required.

[0020] The Second step used in the invention is Identify Agar Plate Orientation (103) uses color segmentation, and all key-points from the previous two parts will finally be sorted from the sorted intersection point, considering the rotation.

[0021] The Final Step in the disclosure is Image registration (104) uses all key points collected so far will be sorted based on the sorted landmark points, taking into account the rotation. With the main points and their corresponding matches, it is now possible to measure the homography such that the image is eventually warped to the predefined reference points, thereby completing the image registration.

[0022] The Invention disclosed here in is validated by taking the Dataset that contains 300 images. The Invention is implemented on the Hardware having the specifications such as Intel 17-5600 2.59GHz, 8 GB DDR3L 1600MHz, and 256 GB SSD having Python 3.8 and the open library, OpenCV 4.2.

BRIEF DESCRIPTION OF DRAWINGS

[0023] The Accompanying Drawings are included to provide further understanding of the invention disclosed here, and are incorporated in and constitute a part this specification. The drawing illustrates exemplary embodiments of the present disclosure and, together with the description, serves to explain the principles of the present disclosure. The Drawings are for illustration only, which thus not a limitation of the present disclosure.

[0024] Referring to Figure 1, illustrates the Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104), in accordance with an exemplary embodiment of the present disclosure.

[0025] Referring to Figure 2, illustrates the Workflow to Identify and Mask the Agar Plate comprising of Color Balance (201), Canny Edge Detection (202), Morphological Operations (203), Border Following (204), Validation (205), RANdom Consensus (206), and ROI Masking (207), in accordance with another exemplary embodiment of the present disclosure.

[0026] Referring to Figure 3, illustrates the Workflow to Identify Compartment Edges comprising of Canny Edge Detection (301), Morphological Operations (302), ROI Masking (303), Hough Transform (304), and Validation (305), in accordance with another exemplary embodiment of the present disclosure.

[0027] Referring to Figure 4, illustrates the Workflow to Identify Agar Plate Orientation comprising of Color Space Segmentation (401), Landmark Points (402), and Sort Key Points (403), in accordance with another exemplary embodiment of the present disclosure.

[0028] Referring to Figure 5, illustrates the Qualitative Results of the Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics, in accordance with another exemplary embodiment of the present disclosure.

[0029] Referring to Figure 6, illustrates the Key and Reference Points Visualization, in accordance with another exemplary embodiment of the present disclosure.

[0030] Referring to Figure 7, illustrates the Image Constructed with Extreme Perspective Distortion before and after Iterations in Registration, in accordance with another exemplary embodiment of the present disclosure.

DETAIL DESCRIPTION OF INVENTION

[0031] Referring to Figure 1, Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104) provides the fundamental image processing steps involved to process the Agar Plate Images for Automatic Diagnostics.

[0032] Referring to Figure 2, the Identify and Mask the Agar Plate (101) is the one of workflow comprising of Color Balance (201), Canny Edge Detection (202), Morphological Operations (203), Border Following (204), Validation (205), RANdom Consensus (206), and ROI Masking (207) for the invention disclosed here.

[0032a] In terms of the illumination or intensity of RGB(Red, Green, Blue) values, different images do not have the same conditions, which means that each image might not get the same result when processed. It is therefore necessary to even out these variables. Images that are too dark or too light do not take full advantage of colour balance, so brightness and strength might also have to be calibrated beforehand. In terms of balancing brightness and contrast, each input will then be processed, followed until further processing by Color Balance (201) using Simplest Color Balance (SCB). The purpose of this step is to enhance the final performance accuracy.

[0032b] The Canny Edge Detection (202) to intensify the edges and reduce any image noise, with contrast, brightness, and colour all balanced. An additional Gaussian blur will first be applied before applying Canny to reduce any bacteria within each compartment. However, configuring Canny with a minimal number of bacteria clusters to the desired level might cause dis-connectivity in the outer contour. The Morphological Operations (203) such as closing would then be used to patch any broken lines.

[0032c] The next is to define any contours. The Circle Hough Transform (CHT) could be used to identify a perfectly circular contour, but due to variations in perspective, the agar plate will always be a bit elliptical. Using CHT, in this case, would give inaccurate results. Therefore, the contour will be found using Border Following (204). The contour found in the highest hierarchy should represent the external contour of the agar plate.

[0032d] In certain cases, due to, for example, bacterial clusters developing close to the edge, parts of the contour may be incomplete. To resolve this question, each image will be validated (205). The contour defined by Border Following will be compared with the approximate contour described by means of CHT. Using Hu Moments the analogy will be made. As the Border Following contour will never be fully circular, as opposed to the CHT contour, a margin of error will be considered. If a value y exceeding the margin of error is returned by the Hu Moment, the entire workflow procedure is repeated. Bacteria clusters covering portions of the agar plate edges in shadowed areas can be the product of invalid matches. By increasing or decreasing the initial brightness or contrast, the problem can be solved. When a valid match is found, the picture is reprocessed.

[0032e] Using RANdom Consensus (206) (RANSAC), the validated (205) contour will then be fitted to an elliptical model, which should provide a better approximation of the external contour, providing the desired key-point range. Finally, a binary mask will be applied to the image with the ROI Masking (207) identified by the contour key-points from past steps, removing the unwanted context.

[0033] Referring to Figure 3, the Identify Compartment Edges (102) is the another workflow of the disclosure comprising of Canny Edge Detection (301), Morphological Operations (302), ROI Masking (303), Hough Transform (304), and Validation (305) to identify the compact edges.

[0033a] The process is the identification of the edges of the compartment and the centre point on the Agar Plate. It is possible to segment each compartment divider as two straight lines, which are compartment edges. Therefore, these shapes can be defined by using Hough Transform. To correctly identify the edges and minimize irrelevant noise that may occur, some pre-processing might be required.

[0033b] Once again, Canny Edge Detection (301) is added to distinguish only the most distinct edges with an extra Gaussian blur. Compared to finding the outer contour, the difference in this case is that only the centre most of the line of the compartment edges is of concern. Each compartment divider consists of two edges, one on each side. It is important to locate the center most line, as the actual centre point of the agar plate is the intersection between the two crossing compartment edges. Using Morphological operation (302) Closing, the edges of each side of each compartment division can be fused to form one thick line. The line can then be reduced to represent a pixel-wide centerline of any divider by using Skeletonize.

[0033c] It will reuse the mask boundaries from the previous segment. The mask's size will be reduced, leaving a new ROI Masking (303). The area beyond the drawn

1U

boundary is not needed to define the positioning and angle of the edges of the compartment. The Hough Transform (304) is used for defining the skeleton as lines with a binary image representing the skeleton of the compartment edges. Lines are sorted and validated (305) in pairs based on the rho and theta values to ensure that only one line is located in each of the dividers' directions.

[0034] Referring to Figure 4, the workflow to Identify Agar Plate Orientation (103) comprising of Color Space Segmentation (401), Landmark Points (402), and Sort Key Points (403) used to provide orientation of Agar Plate Images.

[0034a] Identify plots of their colour space with agar plate orientation. It is important to select the colour space segmented (401) that provides the most visually separable and localized tones of red. The Values from the scatter plot can be read from the colour space chosen to add approximate values to the image.

[0034b] The intersection points between the outer contour and the lines are determined as a first step in locating the landmark points (402) with the chosen compartment colour space segmented. The Pixels will be tested clockwise on a line between each intersection point, creating a rectangular bounding box. It will measure the mean value of the pixel intensity on each line. The one that crosses the segmented compartment should be the line with the highest mean value, and the start and end coordinates of the line would then identify the first two landmark points. In ascending order, the remaining two intersection points will then be ordered clockwise. All key-points from the previous two parts will finally be sorted from the sorted intersection point (403), considering the rotation.

[0035] The Image Registration (104) can be used for Agar Plate Image Registration. Using picture registration, the final touch is to convert the input picture to fit a reference. Since and input image is different, key points are automatically sorted from the previous sections and matched to a reference. Key points representing, in this case, a fully symmetrical agar plate structure, will be identified as the reference. All key points collected so far will be sorted based on the sorted landmark points, taking into account the rotation. With the main points and their corresponding matches, it is now possible to measure the homography such that the image is eventually warped to the predefined reference points, thereby completing the image registration.

[0036] Finally, the elliptical projection of the agar plate needs to be considered in order to boost the accuracy of perspective in more extreme cases. Because the aim is to create a flat projection of the agar plate, additional iterations of the entire process will be used to cope with any residual distortion of perspective.

[0037] Referring to Figure 5, Qualitative Results of the Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Input Image (501), Outer contour of the agar plate identified (502) for the Input Image (501), Compartment Edges identified (503) for the Input Image (501), and The final Output after Registration (504).These Images Illustrates the qualitative results of the invention disclosed here.

[0038] Referring to Figure 6, Key and Reference Points Visualization representing the two images one with Key Points (601) and Reference Points (602). It was possible to align key points with their corresponding reference points for illustrative purposes; the only indicates a total of 37 key points. A total of 10005 key points, divided into 2500 key points per line, 5000 for ellipse and 5 for centre and landmark points.

[0039] Referring to Figure 7, Image Constructed with Extreme Perspective Distortion before and after Iterations in Registration uses number of iteration if perspective distortion still occurring after the first complete iteration, to get promising results. A second iteration produces an Agar Plate with a near-perfect, circular projection. The Image (701) is Image registered without any iteration, (702) is the image after first iteration, and (703) is the promising results of image registration with two iterations.

Claims

DIGITAL IMAGING METHODS AND SYSTEM FOR PROCESSING AGAR PLATE IMAGES FOR AUTOMATED DIAGNOSTICS CLAIMS We claim:

1. Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics comprising of Identify and Mask the Agar Plate (101), Identify Compartment Edges (102), Identify Agar Plate Orientation (103), and Image Registration (104) provides the fundamental image processing steps involved to process the Agar Plate Images for Automatic Diagnostics.

2. Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics as claimed in claim 1, wherein it is used for identify and mask the Agar Plate by Color Balance (201), Canny Edge Detection (202), Morphological Operations (203), Border Following (204), Validation (205), RANdom Consensus (206), and ROI Masking (207).

3. Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics as claimed in claim 1, wherein it uses the Identify Compartment Edges (102) is the another workflow of the disclosure comprising of Canny Edge Detection (301), Morphological Operations (302), ROI Masking (303), Hough Transform (304), and Validation (305) to identify the compact edges.

4. Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics as claimed in claim 1, wherein it uses the workflow to Identify Agar Plate Orientation (103) comprising of Color Space Segmentation (401), Landmark Points (402), and Sort Key Points (403) used to provide orientation of Agar Plate Images.

5. Digital Imaging Methods and System for Processing Agar Plate Images for Automated Diagnostics as claimed in claim 1, wherein it uses Image Registration (104) for generating the Key and Reference Points.