WO2023001940A1

WO2023001940A1 - Methods and systems for generating models for image analysis pipeline prediction

Info

Publication number: WO2023001940A1
Application number: PCT/EP2022/070438
Authority: WO
Inventors: Shih-Jong James Lee
Original assignee: Leica Microsystems Cms Gmbh
Priority date: 2021-07-23
Filing date: 2022-07-21
Publication date: 2023-01-26
Also published as: DE112022003696T5

Abstract

A method is described herein for generating a model for predicting an image analysis parameter for use with an image analysis tool. The method comprises receiving telemetry data associated with image data; and generating a model for predicting an image analysis parameter based on the telemetry data. A further method is described herein for predicting an image analysis parameter for use with an image analysis tool. The method comprising receiving image data associated with a captured image; providing the image data to a model for predicting an image analysis parameter; and outputting the image analysis parameter.

Description

METHODS AND SYSTEMS FOR GENERATING MODELS FOR IMAGE ANALYSIS

PIPELINE PREDICTION

BACKGROUND

[0001] Image analysis tools, or applications, are typically used to improve an acquired image, to identify or classify objects within an image. These image analysis tools may be applied to achieve different image analysis scenarios. The image analysis scenarios may include object analysis and image enhancement, and may be performed in a sequence, or analysis pipeline. However, the inventors have identified that users of such analysis tools may not always obtain the best results, since image analysis tools can be complex and hard to use. Furthermore, researchers dislike repetitive tasks, and are increasingly busy with no time to learn more techniques or tools. Moreover, it can be difficult to automate image analysis process in a reproducible fashion due to heterogeneity. Hence, there is a need to automate image analysis process in a reproducible fashion.

[0002] A problem solved by the present invention is how to provide more automated image analysis and microscopy experiments.

SUMMARY

[0003] In an embodiment of the invention there is provided a computer-implemented method for generating a model for predicting an image analysis parameter for use with an image analysis tool comprising: receiving telemetry data associated with image data; and generating a model for predicting an image analysis parameter based on the telemetry data. For a given input image (e.g. a captured or acquired image), it is possible to predict image analysis parameters from the generated model.

[0004] In a possible implementation of the embodiment, the telemetry data comprises data derived from analysis history associated with an image analysis tool. Accordingly analysis history can be used to train the model.

[0005] In a possible implementation of the embodiment, the data derived from the analysis history comprises an image analysis parameter associated with an image analysis scenario, or a stage, of the image analysis scenario. An image analysis scenario may comprise a sequence of different image analysis stages or steps. [0006] In a possible implementation of the embodiment, the data derived from the analysis history comprises an analysis sequence of at least one image analysis scenario and an image analysis parameter associated with a stage of each of the at least one image analysis scenario. [0007] In a possible implementation of the embodiment, the telemetry data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data. Accordingly data associated with image data (e.g. data of a captured or acquired image) can be used to train the model.

[0008] In a possible implementation of the embodiment, the telemetry data is clustered according to a predetermined similarity criterion to generate a ground truth, and the model for predicting an image analysis parameter is generated based on the ground truth. Accordingly, clustering can be used to generate a ground truth based on the telemetry data, which can be used to train the model.

[0009] In a possible implementation of the embodiment, the method comprising generating a multi-head model for generating a plurality of predicted image analysis parameters.

[0010] In a possible implementation of the embodiment, the method comprising generating a plurality of multi-head models, one for each of a plurality of analysis scenarios.

[0011] In a possible implementation of the embodiment, the model comprises a convolutional neural network.

[0012] In a further embodiment of the invention there is provided a computer-implemented method for predicting an image analysis parameter for use with an image analysis tool comprising: receiving image data associated with a captured image; providing the image data to a model for predicting an image analysis parameter; and outputting the image analysis parameter. Accordingly parameters for subsequent image analysis stages or steps can be generated based on a captured, or acquired, image.

[0013] In a possible implementation of the embodiment, the model is generated according to any one of the methods described above.

[0014] In a possible implementation of the embodiment, the method comprising predicting at least one image analysis scenario using the model, wherein an image analysis parameter is generated for a stage of the at least one image analysis scenario.

[0015] In a possible implementation of the embodiment, the method comprising ranking a plurality of image analysis scenarios, predicted using the model, based on an assessment of scores generated for each of the plurality of analysis scenarios. Accordingly, users can be presented with several analysis scenarios which are ranked to enable users to determine which scenario to select.

[0016] In a possible implementation of the embodiment, the image data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data.

[0017] In a possible implementation of the embodiment, the method comprising applying an image analysis parameter to the captured image according to an image analysis scenario, and outputting modified data.

[0018] In a possible implementation of the embodiment, the method comprising generating an image analysis parameter for an image analysis stage, and providing the generated image analysis parameter after user update to the model for generating a subsequent image analysis parameter.

[0019] In a possible implementation of the embodiment, the method comprising receiving an image analysis parameter from a previous stage in an image analysis pipeline, and providing the image analysis parameter from the previous stage in the image analysis pipeline to the model for generating a subsequent image analysis parameter. An image analysis pipeline may comprise a plurality of image analysis scenarios, stages or steps.

[0020] In a possible implementation of the embodiment, the method comprising generating the image data by concatenating image tiles, generated from the captured image, to form a plurality of image montages, wherein the image tiles comprise critical regions of the image data set determined using intensity and variance thereof. Hence progressive application of the model to an image analysis pipeline improves the prediction of parameters for subsequent stages or steps within the pipeline.

[0021] In a further embodiment of the invention there is provided a model for predicting an image analysis parameter for use with an image analysis tool generated by: receiving telemetry data associated with image data; and generating a model for predicting the image analysis parameter based on the telemetry data.

[0022] In a further embodiment of the invention there is provided a system comprising at least one processor and at least one storage device, wherein the system is configured to perform any one of the methods described above. [0023] In a possible implementation of the embodiment, the system comprising an imaging device coupled to the at least one processor for acquiring microscopy images.

[0024] In a further embodiment of the invention there is provided a computer program with a program code for performing any one of the methods described above.

[0025] In a further embodiment of the invention there is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out any one the methods described above.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present disclosure can be understood with reference to the description of the embodiments set out below, in conjunction with the appended drawings in which:

[0027] FIG. 1 illustrates an overall process flow for an analysis pipeline prediction method for imaging applications in accordance with an embodiment of the invention;

[0028] FIG. 2 illustrates a process flow for generating a model for predicting image analysis parameters in accordance with a further embodiment of the invention;

[0029] FIG. 3 illustrates a method for predicting scenarios and parameters for imaging analysis pipelines in accordance with a further embodiment of the invention;

[0030] FIG. 4 illustrates a method for generating a model for predicting image analysis parameters in accordance with a further embodiment of the invention and a method for predicting an image analysis parameter for use with an image analysis tool in accordance with a further embodiment of the invention; and

[0031] FIG. 5 illustrates an image input and image outputs (FIG. 5A), and a system for performing the methods in accordance with a further embodiment of the invention (FIG. 5B).

DETAILED DESCRIPTION

[0032] The present invention relates to methods and systems for generating models for image analysis pipeline prediction.

[0033] FIG. 1 illustrates the overall process flow for an analysis pipeline prediction method for imaging applications which enable users to apply one or more image analysis scenarios to the captured, or acquired images such as microscopy images. The one or more image analysis scenarios represent processing by utilizing image analysis tools to enhance the images, or can be used for object analysis or detection. For example, the one or more image analysis stages may include 3D object analysis and/or image enhancement, such as denoising. Notably, an image analysis scenario may include, for example, enhancement, detection, measurements, and classification, and a single image analysis pipeline stage may include, for example, 3D, 3D or multi-dimensional object detection/tracking. An image analysis pipeline may include several analysis stages.

[0034] The process flow includes an analysis pipeline 102, data collection portion 104, and prediction learning 106 in a learning portion of the overall process flow. The process flow also includes an analysis pipeline 112, and new data collection 114 in on-going, or progressive, application portion of the overall process flow. The learning portion and the progressive application portion are linked by a prediction application 108, and GUI display application 110. The overall process flow receives, as inputs, user input 116, image data 118, new user data 120 and new image data 122.

[0035] The analysis pipeline 102 receives input data in the form of user input 116, and image data 118 for use in an image analysis scenario. These data are then used to perform one or more image analysis scenarios as is known in the art. As many users perform many image analysis scenarios the data input to the analysis pipeline 102 is collected by the data collection portion 104. The data collection portion 104 is responsible for collecting historical data, or analysis history, in the form of the user input data 116 from past image analysis scenarios which have been performed on the associated image data 118 from the analysis pipeline 102. For example, if the image analysis scenario incudes a 3D object analysis stage, or process, the user may provide several parameters associated with one or more of image smoothing filter size, minimum edge intensity, fill hole size (pm²), object radius (pm), mesh smoothing factor, and minimum edge to centre distance (pm). The data collection portion 104 also receives the image data 118 (i.e. a captured or acquired image) before the analysis is performed to collect and derive data from the image data. This allows the parameter predictions to be performed using the collected data instead of the acquired, or captured, images directly. All collected data are called telemetry data collectively.

[0036] The data collection portion 104 generates telemetry data which is subsequently used for generating a model to enable the prediction of image analysis scenarios and image analysis parameters for use in progressive application portion. The telemetry data includes the analysis sequence (e.g. a sequence of stages or steps) and parameters entered by a user to perform one or more predetermined image analysis stages or steps, such as a 3D object analysis, object tracking, etc.. In addition to the analysis sequence and parameters entered by a user, the telemetry data may also include data associated with any hardware used to capture the image (for example, the hardware specification), user behaviour, user ID, resource usage, and the details of image analysis scenarios commonly used by the user. The telemetry data may also include data associated with the image data (i.e. salient data), such as meta data, and may include image dimensions, image resolution, acquisition parameters and hardware used, and also data derived from the image data such as pixel intensity, variance statistics, and feature maps.

[0037] The telemetry data gathered from the historical or past image analysis scenarios (i.e. analysis history) is then passed to prediction learning 106. Prediction learning 106 is now described in association with FIG. 2.

[0038] FIG. 2 illustrates a process flow for generating a model for predicting image analysis parameters for image analysis scenarios. Telemetry data 202 as described in association with FIG. 1 is acquired which includes one or more parameters associated with one or more image analysis stages of image analysis scenarios, meta data and data derived from captured or acquired images. It will be appreciated that not all of this data will be available and may not be used, but at least the parameters associated with steps of one or more image analysis stages will be typically included in the telemetry data. Furthermore, it will be appreciated that telemetry data 202 includes the telemetry data collected from many tens, hundreds, or even thousands of previously performed image analysis scenarios.

[0039] Pre-processing 204 may be performed on the telemetry data to improve the learning process but also to convert the telemetry data into a form that can be used to train a machine learning model. The pre-processing may include calibration such as converting any lengths to a unit of pixel, converting any areas to a unit of pixel², converting intensity to a value between 0 or 1 depending on the bit depth, and using 1-hot encoding for any category type parameters (e.g. filter type). Other pre-processing steps may be required depending on the form of the data, but these would be apparent to the skilled person in view of the particular data collected and machine learning model being used.

[0040] An additional pre-processing step may also be performed to improve the model generation. As mentioned above, the telemetry data 202 includes data collected from many tens, hundreds, or even thousands of previously performed image analysis scenarios. However, not all of the image analysis scenarios which were previously performed produced a desired result such that it may be appropriate to remove any telemetry data that did not result in a desired result (for example, the post analyzed image did not meet the requirements of the user). One technique to remove any telemetry data that did not result in a desired result is to remove any telemetry data that is associated with an image analysis session where the post analyzed data was not saved. This is considered to be a good proxy for determining if the image analysis scenario did not produce the desired result. For this purpose, the telemetry data would also include an indication as to whether or not a post analysis saving action was performed to allow for the data qualification.

[0041] During data collection 104, an image data set 206 is processed to derive and collect telemetry data 202. In other words for every image, or set of images, in the image data set 206 there will be associated telemetry data. In the example described herein the image data set 206 is a 3D image data set but the image data set 206 may include 2D to ND image datasets. The images in the image data set 206 are sampled to identify critical regions during data collection 104. The critical regions may be identified using pixel intensity and variance, for example. 2D image tiles 208 are identified in each image based on the pixel and intensity values according to predetermined limits as is known in the art. In this example, the image tile 208 is 112x112 pixels. For each image plane in the image data set 206 a montage of 4 tiles is generated to form an image of 224x224 pixels. In one embodiment, a 224x224 pixel image montage is used to derive feature maps for the specific machine learning model described below, but other pixel dimensions may be used if a different machine learning model is used. Furthermore, a plurality of linage montages 210 are typically generated from a 3D instead of 2D image data set (for example, 20 image montages 210 are generated). The image montages are used to derive telemetry data such as feature maps 214 for collection.

[0042] The telemetry data is also used to generate the ground truth such as ideal processing parameters or expected image analysis scenarios for the machine learning process. When generating the ground truth data, the average or mode of the processed telemetry data may be used for a given image data set 206 in simple cases. However, it will be appreciated that this is done for telemetry data from images analyzed under the same image analysis scenario. In a general case, cluster analysis for the telemetric data is performed for each of different image analysis scenarios. To generate ground truth for image analysis scenario, the whole or a subset of telemetry data is used for a cluster analysis. In one embodiment, the accumulated analysis sequences from the analysis history of the telemetry data are used for cluster analysis to generate the analysis scenario ground truth. An analysis sequence could include enhancement stages, object detection stages, object tracking stages, object measurements and object classification stages performed by a user in a session of image analysis scenario. To generate ground truth for image analysis parameter, all or a subset of telemetry data such as processing parameters for an analysis stage and image meta data such as image resolution are used for a cluster analysis to generate the parameter ground truth for an analysis stage of an analysis scenario. Cluster analysis or clustering is the task of grouping a set of samples in such a way that samples in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). Each cluster represents a common mode. The ground truth data for each mode is generated using actual values of a sample from the telemetry data that is closest to a cluster center (not interpolated values) for the machine learning process. For prediction of an analysis scenario, each cluster center can be used as a ground truth scenario. For prediction of processing parameters of an analysis stage, the parameter values of a sample closet to the cluster center can be used as the parameter ground truth of the cluster (group) within an analysis stage. Note that multiple ground truth parameter groups are allowed for an analysis stage. Those ordinary skills in the art should recognize that different clustering algorithms could be used for the telemetry data such as connectivity-based clustering, centroid- based clustering, distribution-based clustering, density-based clustering and grid-based clustering.

[0043] The processed telemetry data 202 (for example, the parameters from historical image analysis scenarios and the ground truth for image analysis parameters such as one or more of image smoothing filter size, minimum edge intensity, fill hole size, object radius, mesh smoothing factor, and minimum edge to centre distance) and the feature map data 214 derived from image montages 210) are passed to the pre-processing 204 and then to machine learning model training 212. In one embodiment of the invention, the feature map data 214 in the example described herein is derived from VGG16 model architecture, but they may also be derived from other convolutional neural network (CNN) architectures such as a SqueezeNet, or a GoogLeNet, etc. The pre-processed telemetry data 204 are used to for machine learning model training 212 in a manner know in the art such as the backpropagation of different CNN architectures using stochastic gradient descent and Adam optimization algorithm, etc.

[0044] Once the machine learning model training 212 has been performed, the model 218 can be used to predict one or more parameters 216 for use in subsequent image analysis steps. It can also be used to predict one or more image analysis scenarios (i.e. several analysis steps or stages). It is envisaged that the machine learning model training 212 will be performed by the tool provider and subsequently deployed on a users’ sites. However, on-going, or progressive, training if desired may still be performed either at the provider or by the user through, for example, federated machine learning.

[0045] In one embodiment, a multi-head model 218 may be generated based on the machine learning model training 212 to generate, or predict, the image analysis parameters 216. In the example described herein, a plurality of image analysis parameters 216 are predicted, and hence the use of a multi-head model 218. However, a different model could be used if only a single parameter is predicted, or if the machine learning model training 212 is structured so as to generate a plurality of image analysis parameters 216. In one embodiment of the invention, each parameter is trained with a CNN model and the CNNs for the parameters are connected by a fully connected output layer so all parameters can be trained together as a set, not individually to yield a multi-head model for multiple parameters. The parameter prediction model for each cluster of the same image analysis stage can be further combined into a multi- head model wherein multiple sets of parameters can be the outputs, each set corresponds to a cluster. In this case, the scores from the multi-head model can be used to rank the parameter sets (i.e. the goodness of the clusters for an input).

[0046] The multi-head model 218 may be structured to generate a single set of image analysis parameters 216, and hence permit a single processed image to be generated. However, the multi-head model 218 may be structured to generate two or more sets of image analysis parameters such that two or more processed images can be generated and a user can decide which one is preferred. For example, the multi-head model 218 may be structured to generate a first set of image analysis parameters 216 to maximize one parameter (e.g. mesh smoothing factor) and a second set of image analysis parameters 216 to maximize a different parameter (e.g. minimum edge to centre distance). The different sets of image analysis parameters can be ranked according to scores from CNN model outputs and previews of processed images from different sets of image analysis parameters may be provided to a user for selection.

[0047] Based on the predicted image analysis parameters 216, the user may improve or update the multi-head model 218, or feedback from the user may be used to improve or update the trained machine learning model 212. This is illustrated by the backward arrows from 216. [0048] Referring again to FIG. 1, the process of on-going, or progressive application is described. In FIG. 1, a prediction application 108 is illustrated which forms part of an image analysis tool. Linked to the prediction application 108 is a GUI display application 110, which presents users with the result of any predictions which have been created using the prediction application 108.

[0049] A user during an application session through analysis pipeline 112 will provide new image data 122, which is used to generate new feature maps and other telemetry data, and new user data 120, such as analysis history will be collected from the users application session progressively including details of one or more desired image analysis steps to be performed on the new image data 122 and the progressive steps in the image analysis scenario. However, it is envisaged that the new user data 120 does not include any progressive (next step) image analysis parameters, since these will be generated by the prediction application 108, as is illustrated by the backward arrow from the GUI display application 110. Of course, a user may enter some of the image analysis parameters, but may still leave some to be predicted.

[0050] The GUI display application 110 subsequently presents the user with a processed image or a preview of a cropped region of the processed image from the prediction application 108, which has been generated using the predicted image analysis parameters 216. In the case of an example 3D object image analysis pipeline stage, the GUI display presents a processed image along with the set of predicted image analysis parameters 216 in the form of image smoothing filter size, minimum edge intensity, fill hole size, object radius, mesh smoothing factor, and minimum edge to centre distance. The user may then decide to adjust the parameters to improve the processed image, which may be referred to as a user update. Such adjustments can be collected and used to further train the models at the user site, or by the tool provider.

[0051] It will be appreciated that the training may not be performed on a per-analysis basis such that on-going, or progressive, data collection will be performed at new data collection 114, which can be subsequently used to perform next round of training after a sufficient volume of telemetry data are collected. However, the progressive application will be carried out sequentially through the different stages of analysis pipeline until all steps of one image analysis scenario are completed. In this progressive application process, new user data 120 will be captured and new image data 122 could also be entered progressively. New telemetry data will be generated and new prediction will be performed and provided to users through GUI display application 110 progressively, stage by stage until the completion of an application session for an image analysis scenario.

[0052] FIG. 3 illustrates a method for predicting scenarios and parameters for imaging analysis pipelines. The methods described in association with FIG. 1 and FIG. 2 have assumed that only a single image analysis scenario is used. However, it is likely that users will want to apply several image analysis scenarios to an image dataset. To achieve this individual machine learning models 218 will be produced for each image analysis pipeline stage and each image analysis scenario. For example, a machine learning model 218 will be produced for 3D object analysis and a machine learning model 218 will be produced for image enhancement for a given image analysis scenario. During the learning processes of such multiple machine learning models 212, any pre-processed telemetry data 204 (or ground truth) is grouped or clustered for the different image analysis scenarios or types. In one embodiment of the invention, as described previously, cluster analysis for the telemetric data is performed for each of different image analysis scenarios. Each cluster represents a common mode. The ground truth data for each mode is generated using actual values of a sample from the telemetry data that is closest to a cluster center (not interpolated parameters) for the machine learning process. This avoids the prediction of parameter sets that are artificial and do not correspond to meaningful analysis results.

[0053] Referring to FIG. 3, user data 310 is provided, which may include the details of the desired image analysis scenarios. A missing profile data unit 302 may be provided with the details of any missing image analysis scenarios based on the commonly used image analysis scenarios, or may simply add the details of all other image analysis scenarios available using mean/median or mode from training set of different scenarios. Telemetry data may also be extracted from user provided image data 316 (for example, a 3D image data set). Meta data 312 and image derived data 314 will also be extracted, as described in associated with FIG. 1 and FIG. 2.

[0054] A scenario and parameter prediction unit 304 receives the data from the missing profile data unit 302, the image data 316, the extracted meta data 312, and the image derived data 314, and generates a predicted scenario(s) 306 and predicted parameters 308.

[0055] The scenario and parameter prediction unit 304 generates the predicted parameters 308 for each of the one or more image analysis scenarios that could be used to generate a processed image for each image analysis scenario, as described herein (i.e. using individually generated models for each image analysis scenario). Each of the processed images are then displayed to the user in the form of predicted scenario(s) 306 to select which scenario and parameter sets are preferred. To facilitate the selection, combinations of image analysis scenarios based on the predicted parameters may also be applied to the image data 316 to provide more processed images.

[0056] Processing may be performed to rank the scenarios by assessing scores generated from the prediction models of the parameter prediction unit 304. The scores can be derived from the confidence of prediction which can be calculated from either machine learning or deep learning models as the methods are well understood in the ordinary skills in the art. If a score is below a predetermined threshold, the processed image, and therefore its associated image analysis scenario or parameter sets, may not be displayed to the user.

[0057] FIG. 4 illustrates a computer-implemented method 402 for generating a model for predicting an image analysis parameter for use with an image analysis tool. The method comprising receiving 404 telemetry data associated with the image data; and generating 406 a model for predicting the image analysis parameter based on the telemetry data.

[0058] FIG. 4 illustrates a further computer-implemented method 408 for predicting an image analysis parameter for use with an image analysis tool. The method comprising receiving 410 image data associated with a captured image; providing 412 the image data to a model for predicting the image analysis parameter; and outputting 414 the image analysis parameter. [0059] FIG. 5A illustrates an input image 502 and two different output images 504 and 506, which have been generated according to the methods described herein corresponding to two different image analysis scenarios. This illustrates different analysis parameters are required to achieve results for different image analysis scenarios.

[0060] FIG. 5B shows a schematic illustration of a system 508 configured to perform a method described herein. The system 508 comprises a microscope 524 and a computer system 514. The microscope 524 is configured to take images and is connected to the computer system 514. The computer system 514 is configured to execute at least a part of a method described herein. The computer system 514 may be configured to execute a machine learning algorithm and to execute image analysis pipeline steps for different image analysis scenarios. The computer system 514 and microscope 524 may be separate entities but can also be integrated together in one common housing. The computer system 514 may be part of a central processing system of the microscope 524 and/or the computer system 514 may be part of a subcomponent of the microscope 524, such as a sensor, an actor, a camera or an illumination unit, etc. of the microscope 524.

[0061] The computer system 514 may be a local computer device (e.g. personal computer, laptop, tablet computer or mobile phone) with one or more processors and one or more storage devices or may be a distributed computer system (e.g. a cloud computing system 522 with one or more processors and one or more storage devices distributed at various locations, for example, at a local client and/or one or more remote server farms and/or data centers). The computer system 514 may comprise any circuit or combination of circuits. In one embodiment, the computer system 514 may include one or more processors 520, which can be of any type.

As used herein, processor may mean any type of computational circuit, such as but not limited to a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processing unit (GPU) 516, a digital signal processor (DSP), multiple core processor, a field programmable gate array (FPGA), for example, of a microscope or a microscope component (e.g. camera) or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 514 may be a custom circuit, an application-specific integrated circuit (ASIC), or the like, such as, for example, one or more circuits (such as a communication circuit) for use in wireless devices like mobile telephones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 514 may include one or more storage devices 610, which may include one or more memory elements suitable to the particular application, such as a main memory in the form of random access memory (RAM), one or more hard drives, and/or one or more drives that handle removable media such as compact disks (CD), flash memory cards, digital video disk (DVD), and the like. The computer system 514 may also include a display device 510, one or more speakers, and input device 512 (e.g. a keyboard and/or controller), which can include a mouse, trackball, touch screen, voice- recognition device, or any other device that permits a system user to input information into and receive information from the computer system 514.

[0062] As used herein the term “and/or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as “/”.

[0063] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0064] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a processor, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some one or more of the most important method steps may be executed by such an apparatus.

[0065] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware, firmware or in software. The implementation can be performed using a non-transitory storage medium such as a digital storage medium, for example a floppy disc, a DVD, a Blu-Ray, a CD, a ROM, a PROM, and EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system locally or in the cloud such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[0066] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0067] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0068] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0069] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer locally or in the cloud.

[0070] A further embodiment of the present invention is, therefore, a storage medium (or a data carrier, or a computer-readable medium) comprising, stored thereon, the computer program for performing one of the methods described herein when it is performed by a processor. The data carrier, the digital storage medium or the recorded medium are typically tangible and/or non-transitionary. A further embodiment of the present invention is an apparatus as described herein comprising a processor and the storage medium.

[0071] A further embodiment of the invention is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example, via the internet.

[0072] A further embodiment comprises a processing means, for example, a computer or a programmable logic device, configured to, or adapted to, perform one of the methods described herein.

[0073] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0074] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0075] In some embodiments, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus or by cloud computing.

[0076] Embodiments may be based on using a machine-learning model or machine-learning algorithm. Machine learning may refer to algorithms and statistical models that computer systems may use to perform a specific task without using explicit instructions, instead relying on models and inference. For example, in machine-learning, instead of a rule-based transformation of data, a transformation of data may be used, that is inferred from an analysis of historical and/or training data. For example, the content of images may be analyzed using a machine- learning model or using a machine- learning algorithm. In order for the machine learning model to analyze the content of an image, the machine-learning model may be trained using training images as input and training content information as output. By training the machine-learning model with a large number of training images and/or training sequences (e.g. processing history profiles, telemetry data) and associated training content information (e.g. ground truth labels or annotations), the machine-learning model "learns" to recognize the content of the images, so the content of images that are not included in the training data can be recognized using the machine-learning model. The same principle may be used for other kinds of sensor data or telemetry data as well: By training a machine-learning model using training sensor or telemetry data and a desired output (parameter set or image analysis scenarios, etc.), the machine-learning model "learns" a transformation between the sensor or telemetry data and the output, which can be used to provide an output based on non-training sensor or telemetry data provided to the machine-learning model. The provided data (e.g. sensor data, telemetry data, meta data and/or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine-learning model.

[0077] Machine-learning models may be trained using training input data. The examples specified above use a training method called "supervised learning". In supervised learning, the machine-learning model is trained using a plurality of training samples, wherein each sampl e may comprise a plurality of input data values, and a plurality of desired output values, i.e. each training sample is associated with a desired output value (ground truth). By specifying both training samples and desired output values, the machine-learning model "learns" which output value to provide based on an input sample that is similar to the samples provided during the training. Apart from supervised learning, semi-supervised learning may be used. In semi- supervised learning, some of the training samples lack a corresponding desired output value. Supervised learning may be based on a supervised learning algorithm (e.g. a classification algorithm, a regression algorithm or a similarity learning algorithm). Classification algorithms may be used when the outputs are restricted to a limited set of values (categorical variables), i.e. the input is classified to one of the limited set of values. Regression algorithms may be used when the outputs may have any numerical value (within a range). Similarity learning algorithms may be similar to both classification and regression algorithms but are based on learning from examples using a similarity function that measures how similar or related two objects are. Apart from supervised or semi-supervised learning, unsupervised learning may be used to train the machine-learning model. In unsupervised learning, (only) input data might be supplied and an unsupervised learning algorithm may be used to find structure in the input data (e.g. by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) so that input values within the same cluster are similar according to one or more (pre-defined) similarity criteria (distance metrics), while being dissimilar to input values that are included in other clusters.

[0078] Reinforcement learning is a third group of machine-learning algorithms. In other embodiments, reinforcement learning may be used to train the machine-learning model. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the taken actions, a reward is calculated. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards).

[0079] Furthermore, some techniques may be applied to some of the machine-learning algorithms. For example, feature learning may be used. In other words, the machine-learning model may at least partially be trained using feature learning, and/or the machine-learning algorithm may comprise a feature learning component. Feature learning algorithms, which may be called representation learning algorithms, may preserve the information in their input but also transform it in a way that makes it useful, often as a pre-processing step before performing classification or predictions. Feature learning may be based on principal components analysis or cluster analysis, for example. Feature learning methods such as deep learning algorithm can learn features without explicit handcrafted features.

[0080] In some examples, anomaly detection (i.e. outlier detection) may be used, which is aimed at providing an identification of input values that raise suspicions by differing significantly from the majority of input or training data. In other words, the machine-learning model may at least partially be trained using anomaly detection, and/or the machine-learning algorithm may comprise an anomaly detection component.

[0081] In some examples, the machine-learning algorithm may use a decision tree as a predictive model. In other words, the machine-learning model may be based on a decision tree. In a decision tree, observations about an item (e.g. a set of input values) may be represented by the branches of the decision tree, and an output value corresponding to the item may be represented by the leaves of the decision tree. Decision trees may support both discrete values and continuous values as output values. If discrete values are used, the decision tree may be denoted a classification tree, if continuous values are used, the decision tree may be denoted a regression tree. A decision tree extension called random forest wherein multiple decision trees are applied and the results are combined in an ensembled decision method.

[0082] Association rules are a further technique that may be used in machine- learning algorithms. In other words, the machine-learning model may be based on one or more association rules. Association rules are created by identifying relationships between variables in large amounts of data. The machine-learning algorithm may identify and/or utilize one or more relational rules that represent the knowledge that is derived from the data. The rules may e.g. be used to store, manipulate or apply the knowledge.

[0083] Machine-learning algorithms are usually based on a machine-learning model. In other words, the term "machine-learning algorithm" may denote a set of instructions that may be used to create, train or use a machine- learning model. The term "machine-learning model" may denote a data structure and/or set of rules that represents the learned knowledge (e.g. based on the training performed by the machine-learning algorithm). In embodiments, the usage of a machine-learning algorithm may imply the usage of an underlying machine-learning model (or of a plurality of underlying machine-learning models). The usage of a machine-learning model may imply that the machine-learning model and/or the data structure/set of rules that is the machine-learning model is trained by a machine-learning algorithm.

[0084] For example, the machine-learning model may be an artificial neural network (ANN). ANNs are systems that are inspired by biological neural networks, such as can be found in a retina or a brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, so-called edges, between the nodes. There are usually three types of nodes, input nodes that receiving input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node may represent an artificial neuron. Each edge may transmit information, from one node to another. The output of a node may be defined as a (non-linear) function of its inputs (e.g. of the sum of its inputs). The inputs of a node may be used in the function based on a "weight" of the edge or of the node that provides the input. The weight of nodes and/or of edges may be adjusted in the learning process. In other words, the training of an artificial neural network may comprise adjusting the weights of the nodes and/or edges of the artificial neural network, i.e. to achieve a desired output for a given input.

A special version of ANN called conventional neural networks (CNN) which is a class of deep neural network. In a CNN, the hidden layers include layers that perform convolutions. Typically this includes a layer that performs a dot product of the convolution kernel with the layer's input matrix. This product is usually the Frobenius inner product, and its activation function is commonly ReLU. As the convolution kernel slides along the input matrix for the layer, the convolution operation generates a feature map, which in turn contributes to the input of the next layer. This is followed by other layers such as pooling layers, fully connected layers, and normalization layers.

[0085] Alternatively, the machine-learning model may be a support vector machine or a gradient boosting model. Support vector machines (i.e. support vector networks) are supervised learning models with associated learning algorithms that may be used to analyze data (e.g. in classification or regression analysis). Support vector machines may be trained by providing an input with a plurality of training input values that belong to one of two categories. The support vector machine may be trained to assign a new input value to one of the two categories. Alternatively, the machine-learning model may be a Bayesian network, which is a probabilistic directed acyclic graphical model. A Bayesian network may represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machinelearning model may be based on a genetic algorithm, which is a search algorithm and heuristic technique that mimics the process of natural selection.

[0086] The invention has been described herein in considerable detail in order to comply with the Patent Statutes and Rules and to provide those skilled in the art with the information needed to apply the novel principles and to construct and use such specialized components as are required. However, it is to be understood that the inventions can be carried out by specifically different equipment and devices, and that various modifications, both as to the equipment details and operating procedures, can be accomplished without departing from the scope of the invention.

[0087] The following is a non-exhaustive list of numbered embodiments which may be claimed:

1. A computer-implemented method for generating a model for predicting an image analysis parameter for use with an image analysis tool comprising: receiving telemetry data associated with image data; and generating a model for predicting an image analysis parameter based on the telemetry data. 2. The computer-implemented method of embodiment 1, wherein the telemetry data comprises data derived from analysis history associated with an image analysis tool.

3. The computer-implemented method of embodiment 2, wherein the data derived from the analysis history comprises an image analysis parameter associated with an image analysis scenario, or a stage of the image analysis scenario.

4. The computer-implemented method of embodiment 2 or embodiment 3, wherein the data derived from the analysis history comprises an analysis sequence of at least one image analysis scenario and an image analysis parameter associated with a stage of each of the at least one image analysis scenario.

5. The computer-implemented method of any preceding embodiment, wherein the telemetry data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data.

6. The computer-implemented method of any preceding embodiment, wherein the telemetry data is clustered according to a predetermined similarity criterion to generate a ground truth, and the model for predicting an image analysis parameter is generated based on the ground truth.

7. The computer-implemented method of any preceding embodiment, comprising generating a multi-head model for generating a plurality of predicted image analysis parameters.

8. The computer-implemented method of any preceding embodiment, comprising generating a plurality of multi-head models, one for each of a plurality of analysis scenarios.

9. The computer-implemented method of any preceding embodiment, wherein the model comprises a convolutional neural network.

10. A computer- implemented method for predicting an image analysis parameter for use with an image analysis tool comprising: receiving image data associated with a captured image; providing the image data to a model for predicting an image analysis parameter; and outputting the image analysis parameter. 11. The computer- implemented method of embodiment 10, wherein the model is generated according to the method of any one of embodiments 1 to 9.

12. The computer-implemented method of embodiment 10 or embodiment 11, comprising predicting at least one image analysis scenario using the model, wherein an image analysis parameter is generated for a stage of the at least one image analysis scenario.

13. The computer-implemented method of any one of embodiments 10 to 12, comprising ranking a plurality of image analysis scenarios, predicted using the model, based on an assessment of scores generated f each of the plurality of analysis scenarios.

14. The computer-implemented method of any one of embodiments 10 to 13, wherein the image data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data.

15. The computer-implemented method of any one of embodiments 10 to 14, comprising applying an image analysis parameter to the captured image according to an image analysis scenario, and outputting modified data.

16. The computer-implemented method of any one of embodiments 10 to 15, comprising generating an image analysis parameter for an image analysis stage, and providing the generated image analysis parameter after user update to the model for generating a subsequent image analysis parameter.

17. The computer-implemented method of any one of embodiments 10 to 16, comprising receiving an image analysis parameter from a previous stage in an image analysis pipeline, and providing the image analysis parameter from the previous stage in the image analysis pipeline to the model for generating a subsequent image analysis parameter.

18. The computer-implemented method of any one of embodiments 10 to 17, comprising generating the image data by concatenating image tiles, generated from the captured image, to form a plurality of image montages, wherein the image tiles comprise critical regions of the image data set determined using intensity and variance thereof.

19. A model for predicting an image analysis parameter for use with an image analysis tool generated by: receiving telemetry data associated with image data; and generating a model for predicting the image analysis parameter based on the telemetry data.

20. A system comprising at least one processor and at least one storage device, wherein the system is configured to perform the method of any one of embodiments 1 to 18.

21. The system of embodiment 20, comprising an imaging device coupled to the at least one processor for acquiring microscopy images.

22. A computer program with a program code for performing the method according to any one of embodiments 1 to 18.

Claims

CLAIMS What is claimed is:

1. A computer-implemented method for generating a model for predicting an image analysis parameter for use with an image analysis tool comprising: receiving telemetry data associated with image data; and generating a model for predicting an image analysis parameter based on the telemetry data.

2. The computer-implemented method of claim 1, wherein the telemetry data comprises data derived from analysis history associated with an image analysis tool, and wherein the data derived from the analysis history comprises an image analysis parameter associated with an image analysis scenario, or a stage of the image analysis scenario.

3. The computer-implemented method of claim 2, wherein the data derived from the analysis history comprises an analysis sequence of at least one image analysis scenario and an image analysis parameter associated with a stage of each of the at least one image analysis scenario.

4. The computer-implemented method of any preceding claim, wherein the telemetry data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data.

5. The computer-implemented method of any preceding claim, wherein the telemetry data is clustered according to a predetermined similarity criterion to generate a ground truth, and the model for predicting an image analysis parameter is generated based on the ground truth.

6. A computer-implemented method for predicting an image analysis parameter for use with an image analysis tool comprising: receiving image data associated with a captured image; providing the image data to a model for predicting an image analysis parameter; and outputting the image analysis parameter.

7. The computer-implemented method of claim 6, wherein the model is generated according to the method of any one of claims 1 to 5.

8. The computer-implemented method of claim 6 or claim 7, comprising predicting at least one image analysis scenario using the model, wherein an image analysis parameter is generated for a stage of the at least one image analysis scenario.

9. The computer-implemented method of any one of claims 6 to 8, comprising ranking a plurality of image analysis scenarios, predicted using the model, based on an assessment of scores generated for each of the plurality of analysis scenarios.

10. The computer- implemented method of any one of claims 6 to 9, wherein the image data comprises at least one of meta data associated with the image data, a feature map of the image data, and data derived from the image data.

11. The computer-implemented method of any one of claims 6 to 10, comprising generating an image analysis parameter for an image analysis stage, and providing the generated image analysis parameter after user update to the model for generating a subsequent image analysis parameter.

12. The computer-implemented method of any one of claims 6 to 11, comprising receiving an image analysis parameter from a previous stage in an image analysis pipeline, and providing the image analysis parameter from the previous stage in the image analysis pipeline to the model for generating a subsequent image analysis parameter.

13. A model for predicting an image analysis parameter for use with an image analysis tool generated by: receiving telemetry data associated with image data; and generating a model for predicting the image analysis parameter based on the telemetry data.

14. A system comprising at least one processor and at least one storage device, wherein the system is configured to perform the method of any one of claims 1 to 12.

15. A computer program with a program code for performing the method according to any one of claims 1 to 12.