WO2022250584A1

WO2022250584A1 - A method and an electronic device for analysing an image

Info

Publication number: WO2022250584A1
Application number: PCT/SE2021/050486
Authority: WO
Inventors: Yohann Morandy
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2021-05-24
Filing date: 2021-05-24
Publication date: 2022-12-01

Abstract

A method performed by an electronic device (111) for identifying a void in a solder joint (610) using image analysis of an image (410) of the solder joint (610) is disclosed. The method comprises:5identifying (103) a solder joint area within at least one portion of the image (410) by detecting a limit of the solder joint (610), the solder joint area being identified as the area inside of the limit, any area outside of the solder joint area being exempted from further analysis,further analysing (104) the solder joint area to detect any void (620) present 10within the solder joint (610) by performing at least one of:- detecting (104a) circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and- analysing (104b) a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one 15image property within the plurality of second regions.

Description

A METHOD AND AN ELECTRONIC DEVICE FOR ANALYSING AN IMAGE

TECHNICAL FIELD

Embodiments herein relate to a method performed by an electronic device for identifying a void in a solder joint using image analysis of an image of the solder joint, and to an electronic device configured to identify a void in a solder joint using image analysis of an image of the solder joint.

BACKGROUND Voiding in the context of soldering in electronics manufacturing is the presence of gaps in solder joints of printed circuit boards (PCBs). Such gaps can result from gassing from the solder paste, flux during reflow, impurities, humidity on the substrate, or from the specific geometry of the PCB, including vias, microvias, etc. Voids cause a lack of mechanical, electrical and thermal contact over the area they cover. This can cause lower reliability, lower thermal performance and to a lesser extent lower current carrying capacity of the affected solder joints. In view of this, industry standards and specific requirements set limits to how much voiding can be tolerated. Voiding is usually measured through X-ray imagery, since X-ray imagery offers the advantage of being able to penetrate the solder joints while being a non-destructive characterization. Algorithms may typically be implemented on the X-ray imagery equipment to automatically detect the voids and estimate the voiding rate. Existing methods for detecting voids and estimating the voiding rate typically involve a lot of user intervention, such as selecting a suitable detection sensitivity and thresholds to use in the image analysis. Adding to this, the detection techniques are typically very simple threshold detection. This makes the results repeatable only within a very narrow range of image contrast, dynamic range, noise pattern, exposure and zoom level. It also makes the results very sensitive to small variations in the settings of the machine when capturing the images. A slight change will result in over- or under-detection even on nearly ideal cases, that is images with very little shadowing and very constant contrast between voids, solder joints, and substrate. As a result, images captured by different machines, or even just in different sessions on the same machine, can typically not be compared with satisfactory precision unless very specific is was taken to obtain a similar degree of detection between different batches of images, which is very difficult in practice. The addition of even minor shadowing due to, for example, components being mounted on top of some solder joints, metallized patterns on layers in front of or behind the joint, or vias, usually makes this kind of analysis less reliable or useless.

SUMMARY

A problem that has been identified by the inventor is that the difficulties associated with low repeatability and sensitivity to small variations in the machine settings lead to extensive manual work in connection with image analysis aiming to detect voids in solder joints. The low detectability using existing thresholding algorithms typically result in that engineers have to manually analyze pictures and identify and mark void limits by hand in order to get good results. This process is of course very wasteful of time and resources and can only be applied to small samples.

An object of embodiments herein is therefore to provide an in at least some aspect improved way of identifying a void in a solder joint using image analysis.

According to an aspect of embodiments herein, the object is achieved by a method for identifying a void in a solder joint according to claim 1.

The method is performed by an electronic device and uses image analysis of an image of the solder joint to identify the void. A solder joint area is identified within at least one portion of the image by detecting a limit of the solder joint, the solder joint area being identified as the area inside of the limit. Any area outside of the solder joint area is exempted from further analysis. The solder joint area is further analysed to detect any void present within the solder joint by detecting circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and/or by analysing a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions. According to another aspect of embodiments herein, the object is achieved by an electronic device according to claim 10.

The electronic device is configured to identify a void in a solder joint using image analysis of an image of the solder joint. The electronic device is configured to identify a solder joint area within at least one portion of the image by detecting a limit of the solder joint, the solder joint area being identified as the area inside of the limit, any area outside of the solder joint area being exempted from further analysis. The electronic device is configured to further analyse the solder joint area to detect any void present within the solder joint by detecting circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and/or analysing a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions.

By first detecting the limits of the solder joint and exempting any area outside of the limits from further analysis, the amount of data to be processed during analysis can be reduced. Furthermore, the removal of data from outside of the detected limits allows adjusting contrast levels, thresholds and similar, so as to improve detectability of voids within the limits of the solder joints. The two complementary ways of detecting voids present within the solder joint furthermore allow detecting voids of irregular shapes in addition to circular voids, which are most commonly present, and the void detectability may thereby be improved. The actual strategy to be used for void detection may be adapted to the use case, that is, the type of solder joint or solder joint pattern which is analysed.

Advantageous embodiments of the method and the electronic device, respectively, are defined in the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS Examples of embodiments herein are described in more detail with reference to attached drawings, in which:

Figure 1 is a flowchart depicting embodiments of a method disclosed herein,

Figure 2 is another flow chart depicting embodiments of a method disclosed herein, Figure 3 is another flow chart depicting embodiments of a method disclosed herein,

Figure 4 is an example image of solder joints used in a method disclosed herein, Figure 5 is another flow chart depicting embodiments of a method disclosed herein,

Figure 6 is an example image of a single solder joint, Figure 7 is another flow chart depicting embodiments of a method disclosed herein,

Figure 8 is another flow chart depicting embodiments of a method disclosed herein,

Figure 9 is another flow chart depicting embodiments of a method disclosed herein, Figure 10 is another flow chart depicting embodiments of a method disclosed herein, and Figure 11a-b schematically illustrate an electronic device according to example embodiments.

DETAILED DESCRIPTION In the following, a method performed by an electronic device for identifying a void in a solder joint using image analysis of an image of the solder joint according to various example embodiments will be described. The method as described herein is particularly applicable to X-ray images, but it may also be applied to images acquired by other characterization techniques, either non-destructive techniques, for example ultrasonic imaging, or destructive techniques including cross-sectional characterization techniques. Thus, it may be applied to any characterization technique that yields an image of a solder joint.

The image of the solder joint may either be a monochromatic image, such as a grayscale image, or it may be a colour image containing two or more colours. It may also be an HSL (hue, saturation, lightness) image or an HSV (hue, saturation, value) image. In all cases, the image is described by at least one image property, which is herein to be understood as a quantifiable property that can be used to describe the appearance of the image. The at least one image property may be at least one of an intensity, a brightness, a lightness, a colour component, a hue, a saturation, and a chromaticity. For monochromatic images, such as X-ray images, the image property is equal to the brightness or the intensity. For colour images, the at least one image property used for analysing the image may be a single image property of the listed properties, or a combination of several image properties.

The method according to embodiments described herein is an algorithm that is designed to be as tolerant to a wide variety of conditions as possible. It is not dependent on a specific machine or brand. The algorithm is primarily intended to be run after image acquisition on an electronic device such as a general-purpose machine, such as a laptop, a server, a mobile device, or using cloud functionality, etc. It may therefore be decoupled from an image acquisition device used to acquire the image and does not need to depend on the acquisition device’s capabilities and does not have to be limited by its restrictions, such as a proprietary image analysis system which cannot be accessed or modified. Moreover, since it can be used on a general purpose machine, image processing does not take up valuable image acquisition time on the image acquisition device. However, the method may also be performed by the image acquisition device if this is deemed suitable. Thus, the electronic device may primarily be a general-purpose machine as described above, or alternatively an image acquisition device.

The algorithm may contain several self-optimizing parts, which aim at automatically determining optimal analysis parameters for each individual image and each individual solder joint within the image. The void identification can thereby proceed with almost zero user interaction and very limited prior knowledge of the image. Initial user input as to which type of solder joint or solder joint pattern is expected may be provided.

The image may typically be a two-dimensional image defined by a horizontal axis and a vertical axis, herein referred to as an x-axis and a y-axis, respectively. The image may of course be a two dimensional slice of a three dimensional image.

A method according to embodiments described herein is schematically illustrated in a flow chart in figure 1. Reference is also made to figure 2, showing a high-level architecture of the method according to embodiments described herein. The method comprises the following actions, which may be taken in any suitable order. Optional actions are referred to as dashed boxes in figure 1. Action 101 - pattern detection

In action 101, a pattern of solder joints within the image may be detected using a thresholding technique. Action 101 may be omitted if the image depicts a single solder joint. The action 101, and actions taken as a part thereof, will be described in further detail with reference to figure 3. The purpose of the optional action 101 of pattern detection to identify a possible regular pattern of solder joints in the image. Since it is usually known beforehand which type of solder joint or solder joint pattern is expected, for example a single solder joint, a few isolated rectangular or pseudo-rectangular joints, or a regular pattern such as a ball grid array (BGA) or a land grid array (LGA), a use case may be defined by a user beforehand, which use case determines whether the action 101 is taken or not. A pattern description including grid positions, such as solder joint offset, size, and pitch, is the result of this action. It is to be noted that the image may not depict an actual solder joint at each identified grid position of the pattern.

Action 102 - image portion identification

Based on the detected pattern, at least one portion of the image, in which an action 103 of identifying a solder joint area is to be performed, is identified. The portion may advantageously be identified as a portion having an extension along the x- and y-axes, which portion is positioned and sized so as to depict a single solder joint. Action 102 may be omitted if the image comprises a single solder joint, in which case the entire image may be used as the at least one portion. At least one image portion is the output of this action.

Action 103 - solder joint area identification

In action 103, a solder joint area is identified within at least one portion of the image by detecting a limit of the solder joint, the solder joint area being identified as the area inside of the limit, any area outside of the solder joint area being exempted from further analysis. A rejection of everything in the image which is not a joint is thereby enabled. This considerably reduces the amount of information to analyse and allows adapting the detection parameters to the properties of individual solder joints rather than to the whole image.

In the action 103 of solder joint area identification, actual solder joints are identified. It is to be noted that not all identified image portions need to comprise a solder joint. The individual solder joint or joints is/are as a part of the solder joint area identification more precisely detected and delimited. Detecting precisely the limits of each joint is necessary for further analysis as well as for accurate solder joint area and thus voiding rate calculation. A solder joint description, including locations and sizes of the actual solder joints within the image, is combined with the pattern description to form a corrected pattern and solder joint description and create solder joint images of the actual solder joints, wherein “empty” portions can be removed.

Action 104 - further analysis to detect voids

In action 104, the solder joint area is further analysed to detect any void present within the solder joint by performing at least one of:

Action 104a: detecting circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and

Action 104b: analysing a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions.

The action 104 is performed for each solder joint image. This is where the actual void detection takes place. As output from the further analysis, void statistics, voiding rate per solder joint and/or global voiding rate, and/or output images, may be provided. Of course, there may be solder joints in which no joint is detected, or in which only one or more circular voids are identified, or in which only one or more arbitrarily shaped voids are identified.

A combination of actions 104a and 104b may advantageously be used to improve the void detection rate. Circular voids are most common and will in an X-ray image typically appear as bright circular regions. Thus, the circular regions detected in action 104a are regions in which the at least one image property deviates from surrounding regions within the solder joint area. This type of detection may be referred to as shape-based detection. Typically, for a monochromatic image such as an X-ray image, brightness or intensity is used to detect the circular voids. As it has been observed that most voids are circular, this shape-based detection usually detects most voids.

The action 104b of analysing the plurality of second regions to identify arbitrarily shaped void(s) may be referred to as region inspection analysis. The region inspection analysis detects arbitrarily shaped voids based on e.g. intensity, contrast and statistical properties of the at least one image property within each one of the second regions. The second region can thus be seen as a detector for detecting arbitrarily shaped voids. Each second region may be in the form of a circular region of a predetermined size, such as a region of a predetermined number of pixels. The second regions should advantageously be defined as regions that are relatively small in comparison to an expected smallest size of the one or more arbitrarily shaped voids, such as smaller than 30%, or 20%, of the expected smallest size of the one or more arbitrarily shaped voids.

The relatively small second region allows making the detection sensitive enough to detect irregularly shaped voids. Using a second region that is too large may cause the detection to miss the outer parts of a void or not be sensitive enough for small voids. On the other hand, a second region that is too small may cause the detection to be less immune to noise and small local variations that are not indicative of voids, resulting in false positive detections.

The action 104 and embodiments thereof will be further described with reference to figure 9.

Action 101 and embodiments thereof

Embodiments of the action 101 of detecting the pattern of solder joints will now be described in greater detail with reference to figure 3, under the assumption that the image is a grayscale X-ray image and that the at least one image property is the intensity of the image. The input image depicting the solder joints to be analysed may first be pre- processed in an action 301, including white or black borders removal, contrast and intensity/brightness adjustments, median filtering and light low pass filtering. This aims at reducing contrast and intensity variations between different images and reducing noise.

The pre-processed image then goes through a pattern detection routine.

The pre-processed image is segmented using two thresholds segmentation, i.e. by an action 302 of defining at least one initial first threshold range for the at least one image property, such as an intensity initial threshold range for the intensity, and rejecting all image data outside of the at least one initial first threshold range. Image data within the initial first threshold range is subsequently used in an action 303 of extracting first horizontal and vertical profiles. Thus, a first threshold range defined by a lower threshold and a higher threshold is defined for the intensity, wherein any pixel having an intensity below the lower threshold or above the higher threshold is rejected. This allows better rejection of interfering elements such as components obscuring solder joints and copper traces. It does however make it more difficult to find proper thresholds. The closer the two thresholds are to each other, the more selective this detection is. This means that more artifacts are rejected, but also that it will be more difficult to find the right thresholds and that there will be a risk of accidentally rejecting part of the joints. Because of this difficulty, and since the optimal thresholds vary from image to image, the thresholds may be chosen automatically through either a closed loop or “quasi closed loop” threshold optimization, as will be explained in more detail below.

The action 101 of detecting the pattern of solder joints may further comprise the action 303 of extracting a first horizontal profile and a first vertical profile of the at least one image property from the image and, based on regularities in the first horizontal and vertical profiles, determining the pattern of solder joints using thresholding in an action 304.

The first horizontal profile and the first vertical profile are simply the average of intensity values of the image along the x- and y-axes as illustrated in figure 4, showing an X-ray image 410, a first horizontal profile 420 and a first vertical profile 430. The X-ray image depicts several solder joints, seen as bright partly shadowed circular regions against a black background. The two-dimensional problem is by using the profiles 420, 430 translated to two coupled one-dimensional problems. The profiles 420, 430 may be filtered using e.g. median filtering and low pass filtering to improve detection reliability. Apart from the apparent simplification, using the profiles 420, 430 instead of the image 410 is also advantageous since it improves the signal-to-noise ratio. This is due to that, for a regular pattern, several solder joints will add up in the profiles, while shadows and other disturbances usually will not. Furthermore, an incomplete solder joint pattern with missing or completely obscured solder joints can still be detected. Thus, problems with shadowing of solder joints may be reduced.

The algorithm in the action 304 tries to find a repeating pattern in the first horizontal and vertical profiles 420, 430. This is done simply using a single threshold detection. Anything in the profiles 420, 430 which is higher than a threshold value is considered a potential region of interest. The threshold value for each profile may be determined through a combination of mean, maximum and minimum value of the profile in order to detect peaks. Some hysteresis may be used in the detection to increase stability. This yields a set of regions of interest in the first horizontal and vertical profiles. These regions of interest may be further filtered by a routine that eliminates small regions and regions for which the distance to neighbouring regions does not approximately fit an average distance between regions. This allows removing irregularly spaced regions.

This ends the pattern detection routine. Its output is the determined pattern, which may be in the form of a set of regions of interest on the x- and y-axes, respectively. Combining this data yields a series of regions of interest in the x-y plane, which regions of interest are candidates for the at least one portion of the image to use in the action 102 of identifying the solder joint area. However, further refining is possible by using a controller that determines whether the determined pattern fulfils certain predetermined conditions, using a closed-loop or a quasi-closed loop as will be explained further ahead.

For further refining, the action of detecting the pattern of solder joints may further comprise an action 305 of comparing the determined pattern of solder joints to a predetermined first model and an action 306 of determining if a predetermined first end condition is fulfilled. If the predetermined first end condition is not fulfilled, the at least one initial first threshold range defined in the action 302 (two thresholds segmentation) is adjusted and the action 101 of detecting the pattern of solder joints is iterated until the predetermined first end condition is fulfilled. When the predetermined first end condition is fulfilled, the determined pattern of solder joints is output in an action 307 and subsequently used to identify the at least one portion of the image, in which the action 102 of identifying a solder joint area is to be performed.

The first model used in action 305 calculates a performance score for the detected pattern using different penalties based on different parameters. The performance score may be used to determine whether the predetermined first end condition is fulfilled. As an example, for a ball grid array, the following penalties may be used: - A “step deviation penalty” estimating how a distance, i.e. step, between regions in x- and y-directions differ. BGAs are assumed to be regular grids and therefore should get very low values (zero for a perfect grid).

- A “radius deviation penalty” estimating how the size (radius) of regions in x- and y- directions differ. BGA joints are assumed to be symmetrical (spheres) and therefore should get very low values (zero for perfect spheres).

- A “radius size penalty” rejecting patterns with balls which are too small to be measured. Patterns with balls large enough to be analysed should get almost zero penalty.

- A “step size penalty” penalizing patterns in which a ration of ball pitch to solder joint radius differs from a predetermined ratio.

- A “grid size bonus” tending to favour large patterns (with many balls) as these are more likely to be the targeted BGA.

- A “pattern score” checking the expected positions of balls on the image for intensity values between the thresholds. It counts the number of pixels within the expected range as 1 and the ones outside as 0. Similarly, it checks the pixels around (outside of) the expected joint position and rates the pixels between the thresholds as 0 (since being outside the expected joint, the intensity should NOT be between the thresholds) and pixels outside of the thresholds as 1. These values are summed and averaged to give a score between 0 and 1.

The total score of the pattern candidate may be calculated as follow: score = 1.0 - stepDeviationPenality*a - radiusDeviationPenality*b - radiusSizePenality*c - stepSizePenality*d + gridScore*e + gridSizeBonus*f; wherein a, b, c, d, e, and fare multiplying factors that may be selected empirically, such as to maximize detection performance in a test set. In a test set according to an example, a = 2, b = 2, c = 1, cf = 0.33, e = 0.25, and f = 1. This score can be used to evaluate the performance of the pattern detection and form a closed loop system when associated with the controller.

The pattern detection routine outputs the detected pattern. The detection performance is evaluated by the above described scoring system, and the controller feeds new thresholds into the pattern detection routine in an iterative process as described above. This is used to optimize the detection performance.

In order to simplify the problem to a single control parameter, the spacing between the lower and higher thresholds of the first threshold range may be chosen to be a constant and the centre value may be used as the control parameter. If more computation power is available, such as in a cloud implementation, both thresholds could be optimised independently to further increase detection performance. However, the relation between the thresholds of the first threshold range and the detection performance may be highly non-linear and non-continuous. This may cause problems for most common controllers such as Proportional-Integral-Derivative (PID) or gradient descent, which get stuck in local maxima. A simple “narrowing down scanning” may therefore be used as the controller with satisfactory performance. This scanning simply chooses regularly spaced test values of the control parameter over the whole range (0-1) and runs the detection. The scan is then repeated with a narrower step around the value which gave the best score in the first scan. This can be further iterated with diminishing returns.

The best scoring detection is then chosen for the next step.

While the pattern detection routine using this “narrowing down scanning” is technically not a closed loop system, it does present a similar advantage of optimizing the detection performance while retaining a deterministic computation time and without the risk of getting stuck in local minima. This is referred to as “quasi closed loop”. The first end condition may in this case be defined so that it is considered fulfilled if a certain percentage of the first threshold range has been scanned through, such as the entire first threshold range. The first end condition may alternatively be defined so that it is considered fulfilled once a certain number of “narrowing downs”, meaning a certain number of times that the step is reduced, or simply once a target score is reached.

If a more traditional controller is used instead of the scanning, a target score and a maximum number of iterations can be set as the predetermined first end condition of the detection loop, and the system will become a truly closed loop. The quasi closed loop system has experimentally been found to give satisfactory performance. While it is never optimal, the absence of the local minima issue is a clear advantage. More optimal closed loop controllers may be found for specific use cases. The pattern detection is tolerant to incomplete patterns, since the first horizontal and vertical profiles will still show regular peaks as long as the pattern is not missing entire rows or columns.

Figure 7 is another flow chart illustrating the action 101 according to example embodiments of the method, in which a closed loop algorithm is used.

Action 103 and embodiments thereof

Embodiments of the action 103 of identifying the solder joint area will now be described in greater detail with reference to figure 5, under the assumption that the image is a grayscale X-ray image and that the at least one image property is the intensity of the image.

The action 103 of identifying the solder joint area may be carried out in a similar way as the action 101 of detecting the pattern of solder joints, but in each of the at least one image portion in which the solder joint area is to be identified. This action may be regarded as a correction action to refine the pattern detected in action 101. The input image portion to be used in the solder joint identification may either be the entire image, in the case of a single solder joint, or it may be a portion of the input image identified in action 102 based on the pattern description from action 101.

Each portion may be rectangular, such as a square of size slightly higher than an estimated diameter of the solder joints. This also means that some portions may not contain any actual solder joint if the pattern is incomplete. If desired, but not necessarily, the portions can be pre-processed. The input image portion depicting the solder joint to be analysed may first be pre-processed in an action 501, including contrast and intensity adjustments, median filtering and light low pass filtering. This aims at reducing contrast and intensity variations between different images and reducing noise. The portions may thereafter be processed by a closed loop or a quasi-closed loop detection system similar to the pattern detection algorithm described above.

The input image portion is thus segmented using two thresholds segmentation, i.e. by an action 502 of defining at least one initial second threshold range for the at least one image property, such as an intensity initial threshold range for the intensity, and rejecting all image data outside of the at least one initial second threshold range. Image data within the initial second threshold range is subsequently used in an action 503 of extracting second horizontal and vertical profiles. This corresponds to the two-thresholds segmentation described in connection with the pattern detection routine.

The action 103 of identifying the solder joint area may further comprise the action 503 of extracting a second horizontal profile and a second vertical profile of the at least one image property from the at least one portion of the image, and, based on the second horizontal and vertical profiles, detecting the limits of the solder joint using thresholding in an action 504. While in this case there is only a single joint to integrate over, just integrating over the width or length of the joint significantly increases the noise immunity and allows better boundary detection. In the case of a circular solder joint, the actual solder joint boundary is not directly detected but instead a horizontal and vertical extension of the solder joint, from which the boundaries are easily inferred.

The second profiles may be normalized to a [0:1] scale, where 0 represents an all low row/column of pixels and 1 an all high row/column of pixels. Each profile may be low pass filtered using a moving average and median filtering to reduce noise. Each profile then goes through a single threshold detection routine which determines which region of the profile is higher than the threshold. The detection threshold may be selected in a loop which starts with a relatively low threshold below 0.5, or below 0.4, or below 0.3. The threshold is then increased until a single large region is detected. This is assumed to be the solder joint. Despite its incremental nature, this loop is fast if implemented using matrix/vector operations.

Remaining regions significantly smaller than the expected size or radius of the solder joint, calculated in the action 101, are rejected. A preliminary solder joint description, comprising the detected limits of the solder joint, is the output of this action.

The action 103 of identifying the solder joint area may further comprise an action 505 of comparing the detected limits of the solder joint to a predetermined second model, and an action 506 of determining if a predetermined second end condition is fulfilled. This may be referred to as a detection performance evaluation. If the predetermined second end condition is not fulfilled, the at least one initial second threshold range defined in the action 502 (two thresholds segmentation) is adjusted and the actions 502-506 of are iterated until the predetermined second end condition is fulfilled. When the predetermined second end condition is fulfilled, the corrected joint description is output in an action 507 and subsequently used to identify the solder joint area used in the action 104 of void detection.

The second end condition may be defined so that it is considered fulfilled based on a comparison result, or if a certain number of iterations have been performed, or if a certain percentage of the second threshold range has been scanned through, such as the entire threshold range.

The second model used herein calculates a performance score for the identified solder joint area using different penalties based on different parameters. The performance score may be used to determine whether the predetermined second end condition is fulfilled, i.e. whether the solder joint detectability is considered sufficient. As an example, for a circular solder joint, the following penalties may be used:

- A “centre deviation from original penalty” that penalizes detections where the new centre position deviates from the centre position detected in action 101. This is because action 103 is only supposed to bring small corrections.

- A “radius deviation penalty” that estimates how the size, such as the radius, of regions in x- and y-directions differ. BGA balls are assumed to be symmetrical (spheres) and therefore should get very low values (zero for perfect spheres).

- A “radius deviation from original penalty” that penalizes detections where the new detected radius deviates from the radius detected in action 101. This is because action 103 is only supposed to bring small corrections. - A “radius size penalty” that simply rejects solder joints which are too small to be measured. Solder joints large enough to be analysed should get almost zero penalty.

- A “filling penalty” that penalises solder joints that are not detected as being “full”, meaning some pixels within the detected area are not between the two thresholds of the second threshold range.

- A “centre bonus” giving an advantage to solder joints which have a high number of pixels between the two thresholds around their detected centre, indicating they are filled and symmetrical. This is done by calculating the horizontal and vertical profiles of the detected solder joint (in a way similar to the image vertical and horizontal profiles described in action 101, where a pixel equals to 1 if between the thresholds and 0 otherwise), and then summing up the number of rows/columns close to the centre which have values over an empirically chosen level.

- A “border penalty”, where the penalty increases for rows/columns far from the centre, which rows/columns have values over the empirically chosen level.

Similarly to action 101, the above defined parameters and multiplying factors may need to be changed for different use cases, although they should suit any circular, square or similar shapes with near 1:1 aspect ratio.

The same “narrowing down scanning” controller may be used as in action 101.

Only solder joints with good detectability will typically be used in the iteration. It is not necessary to use every single solder joint, since the solder joints can be assumed to be similar. It may therefore be advantageous to only use solder joints with high detection score and then use the extracted model for solder joints that cannot accurately be detected. Problems with shadowing of individual solder joints may thereby be reduced.

The solder joint pitch, size (radius in the case of circular joints), and offset as determined in the action 101 are then re-calculated and averaged based on the available data from the solder joints with good detectability in an action 508 of filtering and averaging data from several solder joints with good detectability. Thus, the characteristics of the solder joints with good detectability, including offset from the originally detected pattern, size, and pitch, are averaged to adjust the pattern model. The corrected pattern model is then used, along with the presence check described lower, for the void detection.

These individual joint detections may be used to correct the pattern model but may not be directly used as the final detected positions/descriptions of the solder joints. This is because a single joint detection may easily go wrong and yield slightly false results, however when averaged together the several individual joint detections create a corrected pattern description. When averaged out, individual joint detection errors tend to either cancel each other or be insignificant, such that action results in a better pattern model than the first detection. In particular, for a regular pattern, the averaging action brings a distinct advantage.

This creates a new corrected solder joint and pattern description which is output in an action 509.

A joint presence detection may be ran on all of the solder joints, which simply checks, using the previously found best thresholds and pattern model, if the portion of the image supposedly containing a solder joint has an average value superior to a detection threshold. This is used to detect which positions identified by the pattern description that actually have a solder joint and which ones do not, in the case of incomplete arrays. This separate detection of presence or absence of joints is the reason why it is not necessary to keep every single solder joint during the detection performance evaluation. Even solder joints that were discarded during the detection performance evaluation will be corrected with the parameters detected on clearer solder joints if they are detected as present in the presence detection step.

The solder joint area of each solder joint is identified from the corrected solder joint and pattern description and is output in an action 510 and subsequently used in the action 104 of void detection.

Figure 8 is another flow chart illustrating the action 103 according to example embodiments, in which a closed loop algorithm is used. Action 104 and embodiments thereof

Figure 9 is a flow chart illustrating embodiments of the action 104 of further analysing the solder joint area to detect voids.

Once the accurate presence, position and size of each solder joint has been established, each solder joint is inspected for voids in the action 104, by performing one or both of the actions 104a and 104b. Prior to the void detection routine, the solder joint images, i.e. the at least one portion of the image, may be pre-processed in an action 901 by adjusting contrast and intensity to yield a consistent dynamic range and contrast regardless of their original characteristics. This greatly improves the detection rate and may be done on each individual solder joint independently, as these may have different characteristics even on the same original X-ray image. Any area which is not part of the solder joint is rejected and will not be further processed. Figure 6 is a portion of an image depicting a single solder joint 610, shown as black on a white background, wherein the white background is a result of the rejection of data from areas surrounding the solder joint 610. Two voids 620 are shown as brighter regions within the solder joint 610.

The action 104b of analysing the plurality of second regions may comprise: selecting a centre point around which a first one of the second regions is defined, scanning through the solder joint area by repeatedly selecting a new centre point around which a new one of the second regions is defined, and for each second region, calculating at least one statistical parameter of the at least one image property.

The first centre point may be selected arbitrarily within the solder joint area, or as a predetermined point, such as a point at a predetermined position with respect to a corner of the image. When scanning through the solder joint area, each new centre point may advantageously be selected as a point at a predetermined position with respect to the previous centre point, such as at a predetermined distance along the x- and/or y-axes with respect to the previous centre point. Unless there is prior knowledge of where to expect the voids, the whole joint area needs to be scanned, so usually the first centre point is one corner or top position of the solder joint, and then new centre points are selected throughout the whole joint with a regular step. Smaller steps, ideally 1 pixel, yield better results but require more computational power. The at least one statistical parameter calculated in action 104b may be selected from a minimum value of said at least one image property, a maximum value of said at least one image property, an average value of said at least one image property, a dynamic range of said at least one image property, and a standard deviation of said at least one image property within the second region. Advantageously, two or more statistical parameters are calculated.

The action 104b of analysing the plurality of second regions may further comprise, for each second region: comparing the at least one calculated statistical parameter of the second region to a predetermined range and/or threshold, and based on said comparison, determining if the centre point of the second region is part of a void.

The predetermined range and/or threshold may be selected as a range and/or threshold that was/were empirically found to represent voids in a test set. If the at least one calculated statistical parameter of the second region is determined to be within the predetermined range and/or above/under the threshold, the centre point, i.e. the centre pixel, is considered to be part of a void. If several statistical parameters are chosen, typically all of them should be within their associated range in order for the centre point to be considered as part of a void. If a step of more than one pixel is chosen when moving the centre point in the joint, the centre to be considered as part of a void should advantageously have the same size as the step, in order not to create “gaps” in the void.

It should here be noted that it is only the centre point of the second region which is determined to be part of a void, not the entire second region. This is then repeated using different centre points all over the joint.

The action 104 of further analysing the solder joint area to detect voids 620 may further comprise analysing the solder joint area using region growing, starting from a seed point within a void 620 detected in the action 104a or 104b. The region is grown from the seed point until it is determined to cover the entire void. Irregularly shaped regions may thus be analysed using region growing.

The method as described herein may further comprise the action of, based on the analysed solder joint area, determining a voiding rate in the solder joint. The voiding rate may be determined as a ratio of a total void area to a total solder joint area, either locally within a single solder joint or globally in the image depicting a plurality of solder joints.

If the image 410 is an image of a plurality of solder joints, the action of determining a voiding rate may comprise an action 902 of generating a list of points of the image determined to be part of a void, and aggregating the list of points to an output picture, from which output picture one or more of a local voiding rate in each solder joint, a total voiding rate, a maximum voiding rate, and a minimum voiding rate is/are calculated. Duplicates may be removed as a part of this action.

Region growing may be used on the output picture in order to to determine the size, shape, location, and other characteristics of each individual void within the output picture. This allows getting detailed statistics on the voiding profile in the output picture.

Following the detection of the void, each individual void may now be characterized and the location, size, shape, and other characteristics may be extracted for further individual or statistical analysis.

To further improve the void detection, a closed loop or a quasi-closed loop system similar to the loops used in actions 101 and 103 may be used to dynamically select optimized parameters for the shape and region inspection detection in actions 104a and 104b, respectively. Such an implementation is schematically illustrated in a flow chart in figure 10. This may yield a better void detectability, at the expense of an increased computational complexity. To keep the run time reasonable, a powerful electronic device, a better optimization, an extensive hardware acceleration, or a cloud implementation may be used.

The method disclosed herein may be adapted to a variety of different use cases and material combinations, even though the above described implementation is adapted for ball grid arrays. Such use cases and material combinations for which the method according to embodiments herein has been found to be useful include land grid arrays, steel-steel soldering, steel-aluminium soldering, transistor with solder preform soldering, etc. The general architecture of the method is the same, but geometrical parameters in the detection performance rating need to be changed in action 101 and to a lesser extent in action 103. Action 104 stays mostly unchanged regardless of use case, but detection parameters may need to be adapted to different solder joint profiles in different use cases.

An electronic device 111 configured to identify a void 620 in a solder joint 610 using image analysis of an image 410 of the solder joint 610 is schematically illustrated in figures 11a and 11b.

The electronic device 111 may comprise an input and output interface 1100 configured to communicate with other network entities. The input and output interface 1100 may comprise a wireless receiver (not shown) and a wireless transmitter (not shown).

The electronic device 111 is configured to identify, for example using an identification unit 1110, a solder joint area within at least one portion of the image by detecting a limit of the solder joint, the solder joint area being identified as the area inside of the limit. Any area outside of the solder joint area is exempted from further analysis.

The electronic device 111 is configured to further analyse, for example using an analysing unit 1120, the solder joint area to detect any void present within the solder joint by: detecting circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and/or analysing a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions.

The electronic device 111 may further be configured to: detect, for example by means of a detection unit 1130, a pattern of solder joints within the image using a thresholding technique, and based on the detected pattern, identify the at least one portion of the image in which the solder joint area is to be identified.

The electronic device 111 may be configured to detect the pattern of solder joints by extracting a first horizontal profile 420 and a first vertical profile 430 of the at least one image property from the image 410, and, based on regularities in the first horizontal and vertical profiles 420, 430, determining the pattern of solder joints using thresholding. The electronic device 111 may be further configured to detect the pattern of solder joints by, prior to extracting the first horizontal and vertical profiles 420, 430, defining at least one initial first threshold range for the at least one image property, and rejecting all image data outside of the at least one initial threshold range. Image data within the initial threshold range may subsequently be used for extracting the first horizontal and vertical profiles.

The electronic device 111 may be further configured to detect the pattern of solder joints by comparing the determined pattern of solder joints to a predetermined first model, determining if a predetermined first end condition is fulfilled. If the predetermined first end condition is not fulfilled, the electronic device 111 may be configured to adjust the initial first threshold range and iterating the detection of the pattern of solder joints until the predetermined first end condition is fulfilled.

The electronic device 111 may be configured to identify the solder joint area by extracting a second horizontal profile and a second vertical profile of the at least one image property from the at least one portion of the image, and, based on the second horizontal and vertical profiles, detecting the limits of the solder joint using a thresholding technique.

The electronic device 111 may be configured to analyse the plurality of second regions by selecting a centre point around which a first one of the second regions is defined, scanning through the solder joint area by repeatedly selecting a new centre point around which a new one of the second regions is defined, and for each second region, calculating at least one statistical parameter of the at least one image property.

Optionally, the electronic device 111 may be further configured to analyse the plurality of second regions by, for each second region, comparing the at least one calculated statistical parameter of the second region to a predetermined range or threshold, and, based on said comparison, determining if the centre point of the second region is part of a void 620.

The electronic device 111 may be further configured to, based on the analysed solder joint area, determine a voiding rate in the solder joint 610. Optionally, if the image is an image of a plurality of solder joints 610, the electronic device 111 is further configured to determine the voiding rate by generating a list of points of the image 410 determined to be part of a void 620, and aggregating the list of points to an output picture. From the output picture, one or more of a local voiding rate in each solder joint 610, a total voiding rate, a maximum voiding rate, and a minimum voiding rate is/are calculated.

The embodiments disclosed herein may be implemented through a respective processor or one or more processors, such as a processor 1150 of a processing circuitry in the electronic device 111, together with respective computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the electronic device 111. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the electronic device 111.

The electronic device 111 may further comprise a memory 1160 comprising one or more memory units. The memory 1160 comprises instructions executable by the processor 1150 in the electronic device 111. The memory 1160 is arranged to be used to store e.g. information, indices, channel data, indications, subcarriers, BFWs, data, configurations, and applications to perform the methods herein when being executed in the electronic device 111.

In some embodiments, a computer program 1170 comprises instructions, which when executed by the respective at least one processor 1150, cause the at least one processor 1150 of the electronic device 111 to perform the actions above.

In some embodiments, a respective carrier 1180 comprises the respective computer program 1170, wherein the carrier 1180 is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium. Those skilled in the art will appreciate that the units in the electronic device 111 described above may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in the electronic device 111, that when executed by the respective one or more processors such as the processors described above. One or more of these processors, as well as the other digital hardware, may be included in a single Application-Specific Integrated Circuitry (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a- chip (SoC).

It is also to be noted that the electronic device 111 may be configured to perform the actions according to any one of the embodiments described herein.

When using the word "comprise" or “comprising” it shall be interpreted as non- limiting, i.e. meaning "consist at least of".

The embodiments herein are not limited to the above described preferred embodiments. Various alternatives, modifications and equivalents may be used.

Claims

1. A method performed by an electronic device (111) for identifying a void in a solder joint (610) using image analysis of an image (410) of the solder joint (610), the method comprising: identifying (103) a solder joint area within at least one portion of the image (410) by detecting a limit of the solder joint (610), the solder joint area being identified as the area inside of the limit, any area outside of the solder joint area being exempted from further analysis, further analysing (104) the solder joint area to detect any void (620) present within the solder joint (610) by performing at least one of: o detecting (104a) circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and o analysing (104b) a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions.

2. The method according to claim 1 , further comprising: detecting (101) a pattern of solder joints within the image (410) using a thresholding technique, and based on the detected pattern, identifying (102) the at least one portion of the image (410) in which the action of identifying (103) the solder joint area is performed.

3. The method according to claim 2, wherein the action of detecting (101) the pattern of solder joints comprises: extracting (303) a first horizontal profile (420) and a first vertical profile (430) of the at least one image property from the image, based on regularities in the first horizontal and vertical profiles (420, 430), determining (304) the pattern of solder joints using thresholding.

4. The method according to claim 3, wherein the action of detecting (101) the pattern of solder joints further comprises: prior to extracting (303) the first horizontal and vertical profiles, defining (302) at least one initial first threshold range for the at least one image property, rejecting all image data outside of the at least one initial first threshold range, wherein image data within the at least one initial first threshold range is subsequently used in the action of extracting the first horizontal and vertical profiles, wherein optionally, the action of detecting the pattern of solder joints further comprises: comparing (305) the determined pattern of solder joints to a predetermined first model, determining (306) if a predetermined first end condition is fulfilled, if the predetermined first end condition is not fulfilled, adjusting the at least one initial first threshold range and iterating the action of detecting the pattern of solder joints until the predetermined first end condition is fulfilled.

5. The method according to any one of the preceding claims, wherein the action of identifying (103) the solder joint area comprises: extracting (503) a second horizontal profile and a second vertical profile of the at least one image property from the at least one portion of the image, based on the second horizontal and vertical profiles, detecting (504) the limits of the solder joint (610) using thresholding.

6. The method according to any one of the preceding claims, wherein the action of analysing (104b) the plurality of second regions comprises: selecting a centre point around which a first one of the second regions is defined, scanning through the solder joint area by repeatedly selecting a new centre point around which a new one of the second regions is defined, and for each second region, calculating at least one statistical parameter of the at least one image property, wherein optionally, the action of analysing (104b) the plurality of second regions further comprises, for each second region: comparing the at least one calculated statistical parameter of the second region to a predetermined range or threshold, and based on said comparison, determining if the centre point of the second region is part of a void (620).

7. The method according to any one of the preceding claims, further comprising: based on the analysed solder joint area, determining a voiding rate in the solder joint (610), wherein optionally, if the image (410) is an image of a plurality of solder joints, the action of determining a voiding rate comprises: generating a list of points of the image determined to be part of a void (620), aggregating the list of points to an output picture, from which output picture one or more of a local voiding rate in each solder joint (610), a total voiding rate, a maximum voiding rate, and a minimum voiding rate is/are calculated.

8. A computer program (1170) comprising instructions, which when executed by a processor (1150), causes the processor (1150) to perform actions according to any of the claims 1-7.

9. A carrier (1180) comprising the computer program (1170) of claim 8, wherein the carrier (1180) is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer- readable storage medium.

10. An electronic device (111) configured to identify a void (620) in a solder joint (610) using image analysis of an image (410) of the solder joint (610), the electronic device (111) being configured to: identify a solder joint area within at least one portion of the image (410) by detecting a limit of the solder joint (610), the solder joint area being identified as the area inside of the limit, any area outside of the solder joint area being exempted from further analysis, further analyse the solder joint area to detect any void (620) present within the solder joint (610) by performing at least one of: detecting circular first regions based on a deviation in at least one image property, wherein the circular first regions are identified as circular voids, and analysing a plurality of second regions within the solder joint area to identify one or more arbitrarily shaped voids based on a statistical analysis of the at least one image property within the plurality of second regions.

11. The electronic device according to claim 10, wherein the electronic device (111 ) is further configured to: detect a pattern of solder joints within the image using a thresholding technique, and based on the detected pattern, identify the at least one portion of the image (410) in which the identification of the solder joint area is performed.

12. The electronic device according to claim 11 , wherein the electronic device (111) is configured to detect the pattern of solder joints by: extracting a first horizontal profile (420) and a first vertical profile (430) of the at least one image property from the image (410), based on regularities in the first horizontal and vertical profiles (420, 430), determining the pattern of solder joints using thresholding.

13. The electronic device according to claim 12, wherein the electronic device (111) is further configured to detect the pattern of solder joints by: prior to extracting the first horizontal and vertical profiles (420, 430), defining at least one initial first threshold range for the at least one image property, rejecting all image data outside of the at least one initial threshold range, wherein image data within the initial threshold range is subsequently used in the action of extracting the first horizontal and vertical profiles, wherein optionally, the electronic device (111) is further configured to detect the pattern of solder joints by: comparing the determined pattern of solder joints to a predetermined first model, determining if a predetermined first end condition is fulfilled, if the predetermined first end condition is not fulfilled, adjusting the initial first threshold range and iterating the detection of the pattern of solder joints until the predetermined first end condition is fulfilled.

14. The electronic device according to any one of claims 10-13, wherein the electronic device (111) is configured to identify the solder joint area by: extracting a second horizontal profile and a second vertical profile of the at least one image property from the at least one portion of the image, based on the second horizontal and vertical profiles, detecting the limits of the solder joint using a thresholding technique.

15. The electronic device according to any one of claims 10-14, wherein the electronic device (111) is configured to analyse the plurality of second regions by: selecting a centre point around which a first one of the second regions is defined, scanning through the solder joint area by repeatedly selecting a new centre point around which a new one of the second regions is defined, and for each second region, calculating at least one statistical parameter of the at least one image property, wherein optionally, the electronic device (111) is further configured to analyse the plurality of second regions by, for each second region: comparing the at least one calculated statistical parameter of the second region to a predetermined range or threshold, and based on said comparison, determining if the centre point of the second region is part of a void (620).

16. The electronic device according to any one of claims 10-15, wherein the electronic device (111) is further configured to: based on the analysed solder joint area, determine a voiding rate in the solder joint (610), wherein optionally, if the image is an image of a plurality of solder joints (610), the electronic device (111) is further configured to determine the voiding rate by: generating a list of points of the image (410) determined to be part of a void (620), aggregating the list of points to an output picture, from which output picture one or more of a local voiding rate in each solder joint (610), a total voiding rate, a maximum voiding rate, and a minimum voiding rate is/are calculated.