WO2022074171A1

WO2022074171A1 - System and method for digital image processing

Info

Publication number: WO2022074171A1
Application number: PCT/EP2021/077797
Authority: WO
Inventors: Louis Ryan; Stephen EGAN
Original assignee: Ash Technologies Ltd.,
Priority date: 2020-10-07
Filing date: 2021-10-07
Publication date: 2022-04-14
Also published as: GB202015901D0; EP4226324A1

Abstract

The present invention relates to a digital imaging system and method to automatically determine an optimal focus position with the highest image energy of an object in a digital image, and to automatically inspect and measure heights of objects having varying sizes and relatively smaller relevant features in a digital image. The invention is particularly suitable for use in digital imaging inspection systems for use in inspecting large volume of products produced in a manufacturing process.

Description

Title

SYSTEM AND METHOD FOR DIGITAL IMAGE PROCESSING

Field

The present invention relates to a digital imaging inspection system and method for digital image processing, and more particularly to a system and method to automatically determine an optimal focus position of an object in a digital image.

Background

A digital imaging inspection system such as a digital microscope imaging system uses a digital camera and optics to capture an image of an object of interest. The captured image is generally displayed on a computer monitor for analysis.

Laboratories working in biomedical specialisations such as for example, anatomic pathology, haematology, and microbiology use microscopic digital imaging to examine tissue for the presence and nature of disease. Digital microscope imaging has several advantages which includes the ability to document disease, share findings, collaborate in case of telemedicine, and to analyse morphologic findings. Digital microscope imaging systems can also be used for testing objects in a range of other industries, for example, for testing of medical devices or electronic components or precision engineering objects.

Digital microscope imaging systems are well known in the art. Digital microscope imaging systems with automatic focus means are also known in the art, such as the one disclosed in US6,816,606B2. Such systems can be used in digital image object inspection systems used in manufacturing processes and systems. The known digital microscope imaging systems with automatic focus means are slow and typically allows the system to focus only a fraction of the size of the field of view which reduces the usability of the system, for example it is impossible to inspect objects with varying heights or small relevant features in the Z-axis. A major problem with digital camera microscopes and digital inspection imaging systems is that auto focus that find the best focus for the entire field of view make it almost impossible to inspect an object with varying height components as smaller features will never be in focus.

Other patent publications in the art include WO2019/0373162; WO201 3/116299; US 5,793,900; DE 102004006246 and EP3185213. However all suffer from the same technical problems

Conventional means used for determining image energy leads to an increase in image noise interference and resource usage. Known methods for image energy measurement such as Laplace or direct form folded symmetric FIR’s require more delays and adder trees to be implemented, since such known methods require a full kernel of values to be summed before giving an output.

There is therefore an unfulfilled and unresolved need in the art for a system and method which enables automated inspection and measurement of heights of objects having varying sizes and smaller relevant features in a digital image to overcome at least one of the above mentioned problems.

Summary of Invention

The present invention provides a system and method to automatically determine an optimal focus position of a digital image with the highest image energy while minimizing noise interference, and to automatically inspect and measure heights of objects having varying heights and smaller relevant features in a digital image, as set out in the appended claims. In a preferred embodiment of the present invention, there is provided a digital imaging inspection system, the system comprising an image capturing module configured to capture a plurality of digital images of a plurality of objects; a processor operatively coupled to the image capturing module and having a focus element; at least one filter, and an automated image focusing module; the automated focussing module is configured to determine an optimal focus position for one or more objects based on an image energy output from the filter; and the processor determines the height of at least one object in each of the images from the optimal focus position using a set of known calibrated heights, gathered from interpolating between the focus positions of objects of known heights.

In one embodiment the system comprises an image capturing means and an image processing means operatively coupled to the image capturing means. The image capturing means, for example a digital microscope, comprises an image capturing device, and at least one lens. The image capturing means is configured to capture a plurality of digital images of a plurality of objects which is displayed to a user through a display means. The captured digital image could be for example a live video image or a static image. The lens is positioned along an optical path extending from each of the plurality of objects to the image capturing device.

The image processing means comprises a plurality of focus motors, a comb filter or a blur filter or an energy filter, and an automated focussing means. The automated focussing means is configured to determine the optimal focus position of each of the plurality of captured objects, wherein the optimal focus position has the highest image energy. The image energy is measured and outputted by the comb filter or a blur filter. The automated focussing means is configured to navigate the focus motors to the optimal focus position having the highest image energy based on a positive difference in image energy values between a plurality of consecutive focus positions.

The image processing means is further configured to measure height of each object in each of the plurality of images based on image energy output from the comb filter or a blur filter. The height of each object is determined from the optimal focus position of the object by using a set of calibrated height values obtained by interpolating between the optimal focus positions of a plurality of objects whose heights are predetermined.

In another preferred embodiment of the present invention, a method to determine an optimal focus position having the highest image energy for an object captured in a digital image is provided. The method comprises the first step of initiating an image processing means at an infinite focus position of the object. The image processing means comprises a plurality of focus motors, a comb filter or a blur filter and an automated image focussing means. The image processing means is navigated through a plurality of focus points in various predetermined directions from the infinite focus position. The image energy of each of such plurality of focus points is measured by the comb filter or a blur filter, and the delta or difference in image energy values between consecutive focus points in each of the directions is computed. A positive difference in image energy values between consecutive focus points in a particular direction indicates rising image energy values which further indicates to the image processing means that the optimal focus position with the highest image energy may be in said direction. The direction is then selected for navigating the image processing means to determine the optimal focus position of the object. The optimal focus position of the object is determined based on the image energy at the focus position. The optimal focus position of the object has the highest image energy. In an embodiment of the present invention, a method is provided to determine the height of at least one object captured in a digital image. The height of the object is determined from the optimal focus position of the object, by using a set of calibrated height values obtained by interpolating between the optimal focus position of a plurality of objects whose heights are predetermined. The optimal focus position of the object is determined based on image energy wherein the optimal focus position has the highest image energy. In said embodiment, image energy values are outputted by a blur filter.

The present invention enables a user to automatically focus the system on an area approximately 1/92 (150px x 150px) or an area 0.011 of the size of their field of view. This vastly improves the usability of the system when inspecting an object with varying heights and smaller relevant features. Usage of a comb filter to measure image energy minimizes noise interference compared to a blur filter and ensures predictable and informative image energy profiles across the focus range which in turn enables faster auto focus. Further, usage of the comb filter allows the image processing means to determine the optimal focus position more easily and efficiently as the comb filter gives the automated focusing means enough information to predict the direction to drive the focus motors even if the optimal focus position is within a range of up to thirty five percent (35%) from a reference focus position. Usage of a comb filter also results in lower resource usage since it takes just a horizontal and vertical comb tap.

In one embodiment the image energy outputted by the filter is obtained by adding an input signal with a time delayed version of the input signal, such that constructive interference due to adding of signals indicates a higher image energy output and the automated focussing module subtracts from the captured image only the high energy parts of the image. The present invention hence provides a robust solution for problems identified in the art.

Brief description of drawings

The invention will be more clearly understood from the following description of an embodiment thereof, given by way of example only, with reference to the accompanying drawings, in which:-

Figure 1 is a schematic diagram illustrating a system as per a preferred embodiment of the present invention;

Figure 2 is a graphical representation illustrating minimisation of noise interference due to usage of comb filter for determining image energy, as per a preferred embodiment of the present invention;

Figure 3 is a flow diagram illustrating a method as per a preferred embodiment of the present invention;

Figure 4 is a schematic diagram illustrating a method as per a preferred embodiment of the present invention; and

Figure 5 illustrates the filtering operation of the captured images or video signal using the image processing module as per a preferred embodiment of the present invention.

Detailed description of Drawings

The present invention relates to a system and method for digital image processing, and more particularly to a system and method to automatically determine an optimal focus position of an object in a digital image with the highest image energy, and to automatically inspect and measure heights of objects having varying sizes and relatively smaller relevant features in a digital image. The invention is particularly suitable for used in digital imaging inspection systems for use in inspecting a large volume of products produced in a manufacturing process, such as for example a Printed Circuit Board assembly or medical device or the like. Figure 1 illustrates a schematic diagram of a system for digital image processing for inspection of objects as per a preferred embodiment of the present invention. The system comprises an image capturing means 101 , an image processing means 102, and a display means 103. The image capturing means 101 , for example a digital microscope, comprises an image capturing device 101 a and at least one lens 101 b. The image capturing device 101 a is configured to capture a plurality of images each image having a plurality of objects. The lens is positioned along an optical path extending from each of the plurality of objects captured by the image capturing device 101 a.

The image processing means 102 is operatively coupled to the image capturing means 101 , and the display means 103 is operatively coupled to the image capturing means 101 and the image processing means 102. The display means 103 could be, for example, a monitor of a computing device. The image processing means 102 comprises an automated focusing means 102a, one or more focus motors 102b, a comb filter 102c, and a blur filter 102d. An energy filter can be used to perform the function of the comb filter 102c, and/or the blur filter 102d.The automated focusing means 102a is configured to determine an optimal focus position with the highest image energy for each object captured in an image. The image energy for a plurality of focus points in a plurality of predetermined directions is outputted by the comb filter 102c. Image energy is outputted by the comb filter 102c by adding an input signal with the time delayed version of the same input signal. Such addition causes both constructive interference and destructive interference. When an object is not in focus, the resulting interference is mostly destructive due to which there is a lower energy output. As the object gets more in focus, more of the interference is constructive which in turn gives a higher energy output. This implies that a higher energy value for a focus position means that said focus position is closer to the optimal focus position. The automated focusing means 102a is configured to compute the difference in image energy values between consecutive focus points in each direction. The automated focusing means 102a is further configured to navigate the focus motors 102b in the direction wherein the numerical difference between image energy values of consecutive focus points is positive, based on the prediction that the optimal focus position would lie in said direction since rising image energy values between consecutive focus points or a positive numerical difference between energy values of consecutive focus points would imply that the peak or the optimal focus position having the highest image energy may lie in said direction. The direction of navigation of focus motors 102b is changed if the numerical difference in image energy values between consecutive focus points in such direction is negative. This is because a negative numerical difference would imply that the focus motors are moving away from the optimal focus position. Figure 2 illustrates image energy readings outputted by a comb filter 102c illustrated by reference numeral 200 and a blur filter 102d illustrated by reference numeral 201 , for a reference focus range for a reference object. The output from the comb filter 102c is illustrated in red and the output from the blur filter 102d is illustrated in blue. As shown, noise interference is minimized in case of the comb filter 102c.

Figure 2 further illustrates that the comb filter 102c allows the automated focusing means 102a to determine the optimal focus position more easily and efficiently, as it gives the automated focusing means 102a enough information to predict the direction to drive the focus motors even if the optimal focus position is within a wider range from a reference focus position.

The image processing means 102 is further configured to determine height of each object based on the optimal focus position of the object and an image energy output from the blur filter 102d. The image energy output from the blur filter 102d is used since it provides more pronounced peaks. The height of the object is determined from the optimal focus position of the object by using a set of calibrated height values obtained by interpolating between the optimal focus positions of a plurality of objects whose heights are predetermined.

The automated focussing means 102a comprises an Auto-Focus algorithm that allows a user to focus in on a particular area or object shown in a general field of view of an image in display means 103. The system is configured to allow a user to focus in by clicking or selecting an area of interest of an image shown on the display means 103. The Auto-Focus algorithm can use a feedback loop to find the focus position with the highest image energy. It takes an energy reading at one position, then moves in one direction and takes another reading. The algorithm uses the difference in these readings to decide on a direction, and then drives the focus motor 102b in the desired direction until an energy peak is found. The image energy is calculated using a filter, which minimizes the image noise leading to a more predictable and informative energy profile across the focus range. The filter is preferably a comb filter 102c or a blur filter 102d or a combination of both. It will be appreciated that any type of energy filter can be configured as set out herein to implement the present invention.

The present invention improves the usability of the system when inspecting an object with varying size components or objects in the Z axis. Figure 2 shows the energy readings calculated across the entire focus range when the system was focusing on a test PCB with multiple objects of different heights under inspection. The graph both illustrates the noise elimination and how much further away from the energy peak that a direction can be found. A number of different filtering solutions can be used. With a standard blur filter 201 the focus position needed the peak to be within ~17% of the range from its current position in order to be able to have enough information to predict which direction to drive the focus motors. With the comb filter 200 this was ~35%.

Figure 3 is a flow diagram illustrating a method to determine an optimal focus position having the highest image energy for an object captured in a digital image. The method comprises the step of initiating an image processing means at an infinite focus position of the object 301 . The infinite focus position is chosen since the image processing means has no information initially regarding the focus position of the image. The infinite focus position has most parts of the object in focus. The image processing means is further navigated through a plurality of focus positions of the object in a plurality of predetermined directions from the infinite focus point 302. This step is performed to compute the numerical difference or delta in image energy values between consecutive focus points in each predetermined direction 303.

Figure 4 illustrates determination of the optimal focus position based on the delta or difference in image energy values between consecutive focus positions. As illustrated, the focus position ‘182’ has the highest image energy reading of ‘38155886’. Higher accuracy can be achieved for estimating the optimal focus position by computing the ratio of difference in image energy values between the focus position having the highest reading and the focus positions on either sides of the focus position having the highest reading. Based on this, as illustrated in Figure 4, the optimal focus position is estimated to be at ‘181.875’.

The image energy values are outputted by a comb filter, and a higher value of image energy for a focus position implies that the object is in greater focus at said focus position. Therefore, a positive numerical difference between consecutive focus positions in a particular direction would imply that image io energy is increasing in said direction. Hence, the direction wherein the numerical difference in image energy values between consecutive focus positions is positive is selected 304 for navigating the image processing means.

The image processing means is then navigated in the selected direction to determine the optimal focus position of the object 305. The selected direction is changed if a negative numerical difference in image energy between consecutive focus points is observed, since a fall in image energy implies that the image processing means is moving further away from the optimal focus position. If the rate of increase in image energy increases, or in other words if the positive numerical difference between consecutive focus positions increases, the speed of navigation of the image processing means is also accelerated as this implies that the image processing means is moving in the right direction. This enables the present invention to determine the optimal focus position much faster. Likewise, if the rate of increase in image energy reduces, the speed of navigation of the image processing means is also reduced since this would imply that the peak or the optimal focus position is close to being found. This ensures that the image processing means does not miss out on the optimal focus position and enables determination of the exact optimal focus position. The optimal focus position of the object, having the highest image energy is hence determined 306.

It will be appreciated that the invention enables Z height measurement for individual objects shown on a digital screen. The user can zoom in to max optical zoom and measure the height of a particular component or object . This is done by using the Auto Focus to get the component or object mostly in focus, and then sampling the energy in the nearby range, using the smallest changes in focus motor position, to ensure that the component is in the best possible focus. This function uses a filter, for example a blur filter, to calculate the energy as it provides more pronounced peaks, and the wider area of detection provided by the comb filter is not needed as the object is already substantially in focus.

This focus position is then stored, and converted to a height using a set of stored calibrated heights, gathered from interpolating between the focus positions of objects of known heights.

Sub-step accuracy can also be achieved by looking at the ratio of energy readings either side of the highest reading, as shown in Figure 4.

Figure 5 illustrates the filtering operation of the captured images or video using the image processing module as per a preferred embodiment of the present invention. In this embodiment a blur filter 500 is used. The system takes an input video signal and the filter 500 adds to the input video signal a time delayed version of the same input signal. Constructive interference due to adding of signals indicates a higher image energy output and the automated focussing module subtracts from the captured image only the high energy parts of the image. In other words the invention calculates the high energy from a subtracted blur signal. As shown in Figure 5 part of a sharpening filter a gaussian blur of a video stream is taken and subtracted from the original stream. This subtracted image contains only the high energy parts of the video. This high energy video data is squared and then sent to an area of interest accumulator block 501 to calculate the least mean squared resulting in the image energy. This invention bypasses the need for a Fourier, Sobel or Laplace transforms typically needed to calculate the energy of an image. By tapping part way through a sharpening filter no additional resources are required. It will be appreciated that multiple area of interest generators and implemented so that the accumulators only sum in areas of interest. This can be used to focus on particular objects in the image and the respective heights calculated as outlined above. In an embodiment of the present invention, a method to determine the height of at least one object captured in a digital image. The height of the object is determined from the optimal focus position of the object having the highest image energy, by using a set of calibrated height values obtained by interpolating between the optimal focus position of a plurality of objects whose heights are predetermined. The image energy for this embodiment is deter ined by a blur filter since output from the blur filter provides more pronounced peaks as compared to that from a comb filter. The optimal focus position is determined by navigating an image processing means based on the delta or difference in numerical values of image energies between consecutive focus points in a selected direction.

A direction with consecutive focus points having a positive numerical difference in image energy values is selected. If at any timepoint, a negative numerical difference is observed in image energies of focus points in a selected direction, the selected direction is changed, and the image processing means is navigated in a different direction. The image processing means as per this embodiment comprises a plurality of focus motors, a blur filter, and an automated image focussing means.

Although the present invention has been described with reference to specific embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternate embodiments of the subject matter, will become apparent to persons skilled in the art upon reference to the description of the subject matter. It is therefore contemplated that such modifications can be made without departing from the spirit or scope of the present invention as defined. Further, a person ordinarily skilled in the art will appreciate that the various illustrative method steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and a combination of hardware and software, various illustrations and steps have been described above, generally in terms of their functionality. Whether such functionality is implemented as hardware or a combination of hardware and software depends upon the design choice of a person ordinarily skilled in the art. Such skilled artisans may implement the described functionality in varying ways for each particular application, but such obvious design choices should not be interpreted as causing a departure from the scope of the present invention. The method described in the present disclosure may be implemented using various means. For example, the system described in the present disclosure may be implemented in hardware, firmware, software, or any combination thereof. For a hardware implementation, the processing units, or processors(s) or controller(s) may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.

For a firmware and/or software implementation, software code may be stored in the memory means and executed by a processor. The memory means may be implemented within the processor unit or external to the processor unit. As used herein the term “memory” refers to any type of volatile memory or non-volatile memory. In the specification, the terms "comprise, comprises, comprised and comprising" or any variation thereof and the terms “include, includes, included and including" or any variation thereof are considered to be totally interchangeable and they should all be afforded the widest possible interpretation and vice versa.

The invention is not limited to the embodiments hereinbefore described but may be varied in both construction and detail.

Claims

1 . A digital imaging inspection system, the system comprising: an image capturing module configured to capture a plurality of digital images of a plurality of objects; a processor operatively coupled to the image capturing module and having a focus element, at least one filter, and an automated image focusing module; the automated focussing module is configured to determine an optimal focus position for one or more objects based on an image energy output from the filter; and the processor determines the height of at least one object in each of the images from the optimal focus position using a set of known calibrated heights, gathered from interpolating between the focus positions of objects of known heights.

2. The system as claimed in claim 1 , wherein the image energy outputted by the filter is obtained by adding an input signal with a time delayed version of the input signal, determine a higher image energy output from the added signals and the automated focussing module subtracts from the captured image only the high energy parts of the image.

3. The system as claimed in claim 1 or 2, wherein the automated focussing module is configured to navigate the focussing module to the optimal focus position having the highest image energy based on a positive numerical difference in image energy values between a plurality of consecutive focus positions.

4. The system as claimed in any preceding claim, further comprising a display screen operatively coupled to the image capturing module and the image processing module. The system as claimed in any preceding claim wherein the automated focus module comprises an auto-focus algorithm configured to allow the system to focus in on a particular area or object obtained from at least one digital image showing a plurality of objects. The system as claimed in any of the preceding claims wherein the filter is a blur filter or a comb filter. The system as claimed in claimed in any of the preceding claims wherein the digital image is a live video image or a static image. A method to determine an optimal focus position having the highest image energy for an object captured in a digital image, for use in a digital imaging inspection system, the method comprising the steps of: a) initiating an image processing means at an infinite focus position of the object; b) navigating the image processing means through a plurality of focus positions in a plurality of predetermined directions from the infinite focus position; c) computing the numerical difference in image energy values between consecutive focus positions in each of the plurality of directions; d) selecting a direction for navigating the image processing means wherein the numerical difference in image energy values between consecutive focus positions in the selected direction is positive; e) navigating the image processing means in the direction selected in step (d) to determine the optimal focus position of the object; and f) determining the optimal focus position of the object based on the image energy of the plurality of focus positions in the direction selected in step(d), wherein the optimal focus position of the image has the highest image energy. The method as claimed in claim 8, further comprising the step of increasing the speed of navigation of the image processing means if the positive numerical difference in image energy values between consecutive focus positions increases. The method as claimed in claim 8 or 9, wherein the image processing means comprises a plurality of focus motors, a filter, and an automated image focussing means, and wherein image energy for each of the plurality of focus positions is outputted by the filter. The method as claimed in any of claims 8 to 10, wherein a different direction is selected for navigation of the image processing means if the numerical difference of image energy values between consecutive focus positions in the selected direction is negative. A method to determine the height of at least one object captured in a digital image, for use in a digital imaging inspection system, the method comprising the steps of: a) initiating an image processing means at an infinite focus position of the object in the image; b) navigating the image processing means through a plurality of focus positions in a plurality of predetermined directions from the infinite focus position; c) computing difference in image energy values between consecutive focus positions in each of the plurality of directions; d) selecting a direction for navigating the image processing means wherein the numerical difference in image energy values between consecutive focus positions in the selected direction is positive; e) navigating the image processing means in the direction selected in step (d) to determine the optimal focus position of the object; and f) determining the optimal focus position of the object based on the image energy of the plurality of focus positions in the direction selected in step(d), wherein the optimal focus position of the object has the highest image energy; and g) determining the height of the object from the optimal focus position determined in step (f), by using a set of calibrated height values obtained by interpolating between the optimal focus position of a plurality of objects whose heights are predetermined.

13. The method as claimed in claim 12, further comprising the step of increasing the speed of navigation of the image processing means if the numerical difference in image energy values between consecutive focus positions increases.

14. The method as claimed in claim 12 or 13, wherein the image processing means comprises a plurality of focus motors, a blur filter, and an automated image focussing means, and wherein image energy for each of the plurality of focus positions is outputted by the blur filter.

15. The method as claimed in any of claims 12 to 14, wherein a different direction is selected for navigation of the image processing means if the numerical difference between image energy values between consecutive focus positions in the selected direction is negative.

16. The method as claimed in any of claims 8 to 15 wherein the image is captured using a digital microscope.

19