20220031-02-039062-00155 Methods for Alignment of Sequentially Stained and Consecutive Biological Images Cross-Reference to Related Application [0001] This patent application claims priority to U.S. Provisional Patent Application No. 63/511,499, filed on June 30, 2023, which is herein incorporated by reference in its entirety. Field [0002] The present disclosure relates generally to methods and devices for use in detecting targets in biological tissues aided by imaging, and, in particular, to image alignment. Description of Related Art [0003] Histological specimens are frequently disposed upon glass slides as a thin slice of patient tissue fixed to the surface of each of the glass slides. Using a variety of chemical or biochemical processes, one or several colored compounds may be used to stain the tissue to differentiate cellular constituents, which can be further evaluated utilizing microscopy. Brightfield slide scanners are conventionally used to digitally analyze these slides. [0004] When analyzing tissue samples on a microscope slide, staining the tissue or certain parts of the tissue with a colored or fluorescent dye can aid the analysis. The ability to visualize or differentially identify microscopic structures is frequently enhanced using histological stains. Hematoxylin and eosin (H&E) stains are the most commonly used stains in light microscopy for histological samples. [0005] In addition to H&E stains, other stains or dyes have been applied to provide more specific staining and provide a more detailed view of tissue morphology. Immunohistochemistry ("IHC") stains have great specificity, as they use a peroxidase substrate or alkaline phosphatase ("AP") substrate for IHC stainings, providing a uniform staining pattern that appears to the viewer as a homogeneous color with intracellular resolution of cellular structures, e.g., membrane, cytoplasm, and nucleus. Formalin Fixed Paraffin Embedded ("FFPE") tissue samples, metaphase spreads or histological smears are typically analyzed by staining on a glass slide, where a particular biomarker, such as a protein or nucleic acid of interest, can be stained with
20220031-02-039062-00155 H&E and/or with a colored dye, hereafter "chromogen" or "chromogenic moiety." IHC staining is a common tool in evaluation of tissue samples for the presence of specific biomarkers. In situ hybridization ("ISH") may be used to detect target nucleic acids in a tissue sample. ISH may employ nucleic acids labeled with a directly detectable moiety, such as a fluorescent moiety, or an indirectly detectable moiety, such as a moiety recognized by an antibody which can then be utilized to generate a detectable signal. [0006] Compared to other detection techniques, such as radioactivity, chemo- luminescence or fluorescence, chromogens generally suffer from much lower sensitivity, but have the advantage of a permanent, plainly visible color which can be visually observed, such as with bright field microscopy. However, more substrates with additional properties which may be useful in various applications, including multiplexed assays, such as IHC or ISH assays, are needed. [0007] Additional capabilities for advanced image analysis of histological slides which may be used, for example, in digital pathology may improve detection and assessment of specific molecular markers, tissue features, and organelles, or the like. Achieving high accuracy has been a challenging issue not only for machine learning based algorithms but also for experienced professionals. [0008] One option for increasing accuracy is to use serial sections of tissue, where different biological markers are stained for in each of the two slides, which are then digitalized and aligned, and the staining pattern on one slide may be used to interpret the other slide. One problem with this approach is that the discrete digitized layers or sections may be spatially different (e.g., in terms of orientation, focus, signal strength, and/or the like) from one another. While image alignment may be broadly correct or accurate for larger or more distinct objects (such as bulk tumor detection), it may not have cell-level or organelle-level precision, or the like, since the same cells are not necessarily present on the two slides or images. Similar challenges may arise in analysis of sequentially stained tissue images. This can create challenges for such bio-technological image analysis, which may not be well suited for identifying single cells, organelles, and/or molecular markers, or the like. SUMMARY [0009] According to a first embodiment, a method is provided for aligning a portion of a first image of a tissue with a portion of a second image of the tissue comprising: (a)
20220031-02-039062-00155 obtaining one or more local transformations based on the portion of the first image and/or the portion of the second image, wherein each of the one or more local transformations is: (i) for aligning a patch of the first image to a patch of the second image matching the patch of the first image, and (ii) associated with the patch of the first image and/or with the patch of a second image; and (b) determining a transformation for aligning the portion of the first image to the portion of the second image according to the one or more local transformations. [0010] According to a second embodiment, in addition to the first embodiment, said obtaining the one or more local transformations is performed by selecting the one or more local transformations out of a set of local transformations. [0011] According to a third embodiment, in addition to the second embodiment, each transformation out of the set of local transformations is stored in association with a location within the first image and/or the second image for which said local transformation has been derived. [0012] According to a fourth embodiment, in addition to the third embodiment, said selecting the one or more local transformations includes selecting each of the one or more local transformation(s) based on a condition. The condition(s) may require, e.g., that: (i) a location associated with said local transformation is in a distance relative to the portion of the first image smaller than a neighborhood threshold; and/or (ii) the location associated with said local transformation is in a distance relative to the portion of the second image smaller than the neighborhood threshold. [0013] According to a fifth embodiment, in addition to the third or fourth embodiment, said selecting the one or more local transformations comprises selecting one local transformation of which the associated location is closest to the portion of the first image and/or to the portion of the second image; and the transformation for aligning the portion of the first image to the portion of the second image is determined to be said one local transformation. [0014] According to a sixth embodiment, in addition to the third or fourth embodiment, said selecting the one or more local transformations is selecting of K local transformation of which the associated location is closest to the portion of the first image and/or to the portion of the second image; K is larger than 1 and smaller than or equal to the number of the local transformations in the set of local transformations; and the transformation for aligning the portion of the first image to the portion of the second image is determined to be a function of the K local transformations.
20220031-02-039062-00155 [0015] According to a seventh embodiment, in addition to the sixth embodiment, said function is a weighted average and the weights are determined according to a distance between the location associated with the respective transformation and the location of the portion of the first image and/or the portion of the second image. [0016] According to a eighth embodiment, in addition to the third or fourth embodiment, said selecting the one or more local transformations is selecting of K local transformation individually for each sample out of a set of samples included in the portion of the first image; K is larger than 1 and smaller than or equal to the number of the local transformations in the set of local transformations; and a transformation for aligning said sample of the portion of the first image to a sample of the portion of the second image is determined to be a function of the K local transformations. [0017] According to a ninth embodiment, in addition to the eighth embodiment, said function is a weighted average and the weights are determined according to a distance between the location associated with the respective transformation and the location of said sample. [0018] According to a tenth embodiment, in addition to the eighth or ninth embodiment, transformations for aligning respective samples belonging to the portion of the first image but not included in the set of samples are determined as a function of a one or more transformation determined for respective one or more samples in the set of samples. [0019] According to a eleventh embodiment, in addition to any of the first to tenth embodiment, the portion of the first image and/or the portion of the second image is a field of view, FOV, of the first image and/or the second image; and the FOV is selected by a user. [0020] According to a twelfth embodiment, in addition to the eleventh embodiment, the FOV is selected by using a graphical user interface, GUI. [0021] According to a thirteenth embodiment, in addition to any of the eleventh to twelfth embodiment, the method is further comprising: upon selection of a new FOV, automatically perform said determining the transformation for aligning the FOV of the first image to the FOV of the second image according to the one or more local transformations, and align the FOV of the first image to the FOV of the second image. [0022] According to a fourteenth embodiment, in addition to the thirteenth embodiment, said determining the transformation is determining of a transformation common to all samples of the new FOV.
20220031-02-039062-00155 [0023] According to a fifteenth embodiment, in addition to the thirteenth embodiment, said determining the transformation is determining of a transformation common for a part of the new FOV that was not part of the preceding FOV. [0024] According to a sixteenth embodiment, in addition to any of the eleventh to fifteenth embodiment, the method including displaying the first image and/or second image, wherein the selection of the new FOV is performed by moving the displayed first image and/or second image including panning, zooming in/out, and/or moving to a pre-selected coordinate. [0025] According to a seventeenth embodiment, in addition to any of the second to sixteenth embodiment, the method comprising determining the set of local transformations including: (i) obtaining a set of pairs of matching key points in the first image and the second image; (ii) for each pair out of the set of pairs of matching key points: determining a local transformation for aligning a first patch located at the key point of said pair in the first image with a second patch located at the key point of said pair in the second image, and including the local transformation into the set of local transformations. [0026] According to an eighteenths embodiment, in addition to the seventeenth embodiment, the pairs in the set of pairs of the matching key points are obtained in the first image in a first resolution and in the second image in the first resolution. Said determining a local transformation comprises: (i) extracting the first patch located at the key point of said pair in the first image in a second resolution, and (ii) extract the second patch located at the key point of said pair in the second image in the second resolution. The second resolution is higher than the first resolution. [0027] According to a nineteenth embodiment, in addition to the eighteenths embodiment, said determining a local transformation further comprises determining pairs of patch key points matching in the first patch and in the second patch, and derive said local transformation according to on the pairs of patch key points as a Euclidean transformation. [0028] According to a twentieth embodiment, in addition to any of the seventeenth to nineteenth embodiment, said obtaining a set of pairs of matching key points in the first image and the second image comprises: obtaining a superset of pairs of matching key points; selecting, among the pairs of matching key points of the superset, the best matching pair of key points; moving the best matching pair of key points from the superset to said set; and repeating a plurality of times the steps of: (i) selecting, among
20220031-02-039062-00155 the pairs of matching key points of the superset, the next best matching pair of key points that fulfills a predefined condition, wherein the predefined condition includes a condition according to which the next best matching pair of key points is to be located farther than a threshold distance from any of pairs of key points moved to said set; and (ii) moving the next best matching pair of key points from the superset to said set. [0029] According to a twenty-first embodiment, in addition to any of the seventeenth to nineteenth embodiment, said obtaining a set of pairs of matching key points in the first image and the second image comprises receiving at least one or more of the a set of pairs of matching key points from a user. [0030] According to a twenty-second embodiment, in addition to the twenty-first embodiment, the receiving the at least one out of the set of pairs of matching key points from a user includes: (i) displaying the first image and the second image so that the first image is pre-aligned with the second image; (ii) providing a user a graphical user interface for marking at least one position on the displayed first image and the second image; (iii) detecting the at least one positions within the displayed first image and second image marked using the interface; and (iv) based on the detected at least one position, determine the at least one out of the set of pairs of matching key points. [0031] According to a twenty-third embodiment, in addition to any of the seventeenth to nineteenth embodiment, said obtaining a set of pairs of matching key points in the first image and the second image comprises: dividing at least a portion of the first and/or second image into a pattern of image units; and determining the set of pairs of matching key points to include one pair of matching key points per image unit. [0032] According to a twenty-fourth embodiment, in addition to any of the second to twenty-third embodiment, said determining the set of local transformations includes: determining pairs of matching portions of the first image in the second image based on a similarity of image intensities between the first image and in the second image; deriving, for at least one of the pairs of matching portions, a local transformation for aligning portion of said at least one of the pairs in the first image to the matching portion of said at least one of the pairs in the second portion; and including the local transformation into the set of local transformations. [0033] According to a twenty-fifth embodiment, in addition to any of the second to twenty-fourth embodiment, said determining the set of local transformations includes providing a user with a graphical user interface allowing a user to: (i) display the first image and/or the second image in a selectable resolution, (ii) apply at least one of
20220031-02-039062-00155 desired rotation, translation, and scaling to the displayed first image and/or second image, and (iii) add a local transformation resulting from the at least one of desired rotation, translation, and scaling into the set of local transformations. [0034] According to a twenty-sixth embodiment, in addition to any of the second to twenty-fifth embodiment, said determining a transformation for aligning the portion of the first image to the portion of the second image: is determining of a global transformation for aligning the first image to the second image, and includes calculating a function of the local transformations in the set of local transformations, wherein the local transformations are represented by matrices. [0035] According to a twenty-seventh embodiment, in addition to the twenty-sixth embodiment, said function is an average. [0036] According to a twenty-eighth embodiment, in addition to any of the seventeenth to twenty-seventh embodiment, in said determining the set of local transformations, local transformations that include a rotation by more than a threshold angle are not included in the set; and the threshold angle is larger than 0. [0037] According to a twenty-ninth embodiment, in addition to any of the first to twenty- eighth embodiment, the method comprising, before obtaining said one or more local transformations: scaling of the first image to match a ratio between a unit of length and a sample size in at least one direction; the at least one direction being vertical and/or horizontal direction, and/or transforming the first image including flipping the first image around a vertical axis and/or flipping the first image around a horizontal axis. [0038] According to a thirtieth embodiment, in addition to any of the first to twenty-ninth embodiment, the method comprising, before obtaining said one or more local transformations: distinguishing, in the first image, a foreground from a background, detecting, within the foreground, a largest connected region, and perform a rough alignment of the first image to the second image in a third resolution lower than the full resolution of the first image by aligning the largest connected region to the second image. [0039] According to a thirty-first embodiment, in addition to the thirtieth embodiment, the distinguishing the foreground from the background is performed by an entropy- threshold-based method which outputs a bitmap indicating, for each unit of the first image, whether said unit belongs either to the background or to the foreground. [0040] According to a thirty-second embodiment, in addition to thirtieth or thirty-first embodiment, said rough alignment of the largest connected region to the second
20220031-02-039062-00155 image comprises: (i) detecting, in the foreground of the first image in said third resolution, key points and feature descriptors associated respectively with the key points; (ii) detecting, in the second image in said third resolution, key points and feature descriptors associated respectively with the key points; (iii) obtain rough-resolution pairs of key points by matching the key points and the feature descriptors detected in the first image with the key points and the feature descriptors detected in the second image; (iv) select a subset of the rough-resolution pairs of key points with the best matching; and (v) derive a rough transformation for said rough alignment according on the subset of the pairs of key points. [0041] According to a thirty-third embodiment, in addition to thirty-second embodiment, the rough transformation is an Euclidean transformation. [0042] According to a thirty-fourth embodiment, in addition to any of the first to thirty- third embodiment, the first image and the second image are images of the tissue, sequentially scanned and/or differently stained. [0043] According to a thirty-fifth embodiment, in addition to any of the first to thirty- fourth embodiment, the method comprises, after applying said transformation for aligning the portion of the first image to the portion of the second image, applying an additional alignment by optical flow to the first image. [0044] According to a thirty-sixth embodiment, a computer program is provided stored on a non-transitory medium and comprising code instructions which, when executed on one or more processors perform the steps of the method according to any of the first to thirty-fifth embodiment. [0045] According to a thirty-seventh embodiment, an apparatus is provided for aligning a portion of a first image of a tissue with a portion of a second image of the tissue comprising processing circuitry configured to: obtain one or more local transformations based on the portion of the first image and/or the portion of the second image, wherein each of the one or more local transformations is: (i) for aligning a patch of the first image to a patch of the second image matching the patch of the first image, and (ii) associated with the patch of the first image and/or with the patch of a second image; and determine a transformation for aligning the portion of the first image to the portion of the second image according to the one or more local transformations. [0046] According to a thirty-eighth embodiment, the apparatus further comprising: a storage device storing the one or more local transformations, an input module configured to obtain the portion of the first image and/or the portion of the second
20220031-02-039062-00155 image; and an output module configured to output images including the portion of the first image and/or the portion of the second image, wherein the processing circuitry is configured to obtain one or more local transformations from the storage device. [0047] According to a thirty-ninth embodiment, in addition to the thirty-seventh or thirty- eighth embodiment, the input module includes at least one key and/or a touch screen and/or a voice input device; and the output module includes a screen and/or the touch screen. [0048] According to a fortieth embodiment, in addition to the thirty-ninth embodiment, the portion of the first image and/or the portion of the second image is a field of view, FOV, of the first image and/or the second image; and the input module is configured to enable a user to select the FOV. [0049] According to a forty-first embodiment, in addition to the fortieth embodiment, the input module and the output module are at least in part implemented by a graphical user interface, GUI. [0050] According to a forty-second embodiment, in addition to the fortieth or forty-first embodiment, the processing circuitry is configured to: upon selection of a new FOV by a user over the input module, automatically perform said determining the transformation for aligning the FOV of the first image to the FOV of the second image according to the one or more local transformations, and obtain an aligned FOV of the first image by aligning the FOV of the first image to the FOV of the second image; and the output module is configured to output the aligned FOV. [0051] According to a forty-third embodiment, in addition to any of the fortieth to forty- second embodiment, the output module is configured to display the first image and/or second image, the input module is configured to receive an input from the user for said selecting of the new FOV by moving the displayed first image and/or second image including panning, zooming in/out, and/or moving to a pre-selected coordinate. BRIEF DESCRIPTION OF DRAWINGS [0052] A further understanding of the nature and advantages of particular embodiments may be realized by reference to the remaining portions of the specification and the drawings, in which like reference numerals are used to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one or multiple similar components. When reference is made to a
20220031-02-039062-00155 reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components. [0053] FIG.1 is a flow diagram illustrating a method for aligning a portion of a first image of a tissue with a portion of a second image of the tissue. [0054] FIG.2 is a schematic drawing illustrating an image, a view within an image and image patches. [0055] FIG.3 is a block diagram illustrating an exemplary structure of a device for aligning a portion of a first image of a tissue with a portion of a second image of the tissue. [0056] FIG.4 is a block diagram illustrating a functional structure of a processing circuitry. [0057] FIG.5 is a schematic drawing illustrating an image of a tissue and a field of view at a certain location within the image. [0058] FIG.6 is a schematic drawing illustrating an image of a tissue including anchor points associated with respective local transformations and the field of view. [0059] FIG.7 is a schematic drawing illustrating a locality condition for selecting a set of local transformations to be used for alignment of the field of view. [0060] FIG.8 is a schematic drawing illustrating a determination of alignment transformations for parts of the field of view individually. [0061] FIG.9 is a schematic drawing illustrating determination of an alignment transformation as the local transformation of the closest anchor point. [0062] FIG.10 is a flow diagram illustrating an exemplary method for determining local transformations using key points. [0063] FIG.11 is a flow diagram illustrating an exemplary method for obtaining a set of key points for use in determining the local transformations. [0064] FIG.12 is a block diagram illustrating an exemplary system comprising a device for performing alignment of two images and some peripheral or external devices. [0065] FIG.13 is a view of an exemplary graphical user interface (“GUI”) enabling user to initiate or control the automated image alignment. [0066] FIG.14 is a view of an exemplary GUI enabling a user to select pairs of matching key points. [0067] FIG.15 is a view of an exemplary gGUI enabling a user to select local transformations.
20220031-02-039062-00155 [0068] FIG.16 is a schematic drawing illustrating image portion subdivided into tiles for which the respective key points are determined. [0069] FIG.17 is a flow diagram illustrating steps of an exemplary pre-processing workflow. [0070] FIG.18 is a schematic drawing illustrating an exemplary screenshot showing a dynamic alignment upon selection of a new view. [0071] FIG.19 is a schematic drawing illustrating an exemplary screenshot showing a persistent alignment upon selection of a second, new view. [0072] FIG.20 is a schematic drawing illustrating an exemplary screenshot showing a persistent alignment upon selection of a third view. [0073] FIG.21 is a schematic drawing illustrating an exemplary set-up for obtaining consecutive slides. [0074] FIG.22 is a schematic drawing illustrating an example of a local alignment visualization. DESCRIPTION [0075] The present disclosure relates generally to methods and devices for use, for example, in multiplexed assays or in cases of consecutive slides that may be observed together for detecting target molecules or other parts of biological tissue. Such methods have a wide utility in diagnostic applications, in choosing appropriate therapies for individual patients, or in training neural networks or developing algorithms for use in such diagnostic applications or selection of therapies. The present disclosure may also facilitate implementing annotation data collection and autonomous annotation, and, more particularly, implementing sequential imaging of biological samples for generating training data for developing deep learning based models for image analysis, for cell classification, for feature of interest identification, and/or for virtual staining of biological samples. [0076] Joint analysis of multiple scanned images of pathological stained tissue from a single block are often required for clinical diagnosis, using multiple stains (e.g., H&E, different IHCs or the like, as mentioned above) and stain modalities (e.g., brightfield, fluorescence or the like). Moreover, the development of machine learning and artificial intelligence models for analyzing scanned tissue samples can also require the use of images involving multiple stains and stain modalities. Such multiple images may be
20220031-02-039062-00155 scans of consecutive sections of the same tissue block or scans of sequential staining of the same tissue section, or both. [0077] A preliminary step for enabling such joint analysis is aligning the images or image portions such that corresponding parts of the tissue are situated as near to each other as possible when the images are put on the same reference frame. Such alignment entails finding a transformation indicating how the images should be rotated, translated, and possibly scaled, so that they would overlap in a possibly suitable (ideally optimal) way. Finding a reasonable global alignment between consecutive sections and/or different stains and stain modalities is a challenging problem. Manual alignment is cumbersome and requires a human user to identify corresponding parts of the tissue “by eye”. Automated alignment is also challenging. When aligning scans of consecutive sections, finding a global alignment which perfectly aligns all local parts of the tissue may not be generally possible, since tissue structures on all scales may change from section to section. Even for sequentially stained sections, finding a perfect global alignment is often impossible, due to differences in the presentation of different stains or staining modalities, as well as due to possible deformations and artifacts introduced in the process of sequential staining. [0078] As a result, even when a globally reasonable alignment would be found, some misalignments presented when looking at specific regions of interest may still be too great to allow for an accurate analysis. For a joint analysis by a pathologist, usually a cell-level or a nucleus-level alignment is sufficient. However, for sequentially stained images used in the development of machine learning models, the level of alignment needed may sometimes (e.g., virtual staining) be more stringent, requiring a pixel- perfect alignment between the images. [0079] According to an aspect, a method is provided for aligning a portion of a first image of a tissue with a portion of a second image of the tissue. Such method is illustrated in Fig.1. [0080] Alignment or aligning refers to a process of transforming one image (the first image) such that its content is aligned with a content of another image (the second image). The first image will be also referred to as a “moving image” because it may be moved (transformed) for the purpose of the alignment, whereas the second image will be also referred to as a “fixed image”, because it serves as a reference for aligning the first, moving image. The fixed image may be understood as a reference image and the mobile image may be understood as image to be aligned to the reference image.
20220031-02-039062-00155 [0081] The alignment is not necessarily performed for the entire first image or the entire second image that are available and have been captured by an imaging device. Rather, the alignment may be performed for a portion of the first image and a respective portion of the second image. It is noted that the term “portion” herein refers to any portion of the image (subset of image samples) or to the entire image. One example of a portion is a field or view (FOV). In general, the FOV is a part of a tissue scan being viewed or accessed. The FOV may be viewed, for example, by a trained professional using a corresponding device for viewing images such as a computer with a display. Since tissue images may have a high resolution, they can be viewed applying various different zooming factors, so that only a part of the image content (e.g. the tissue) may be seen on the display at a time. The FOV may be accessed, for example, by a computer or another device reading the FOV from a computer readable medium such as a memory or another kind of storage. The FOV may be also referred to herein as “view”. A FOV of an image may be defined by specifying a combination of a location, a size and/or magnification (resolution level). In a context of viewing aligned images, a view can be advantageously defined in the coordinate reference frame of the fixed image (second image). [0082] The method for aligning shown in Fig.1 comprises a step 110 of obtaining (e.g. by computing, by fetching from a storage and/or by human-machine interaction) one or more local transformations and a step 120 of determining a transformation for aligning the portion of the first image (moving image) to the portion of the second image (fixed image) according to the one or more local transformations. The method may further comprise a step 130 of aligning the portion of the first image (moving image) to the portion of the second image (fixed image) including applying said transformation for aligning (referred to in Fig.1 as an alignment transformation). [0083] The one or more local transformation are determined in step 110 based on the portion of the first image and/or the portion of the second image. Each of the one or more local transformations serves for aligning a patch of the first image to a patch of the second image matching the patch of the first image. Moreover, each of the one or more local transformations is associated with the patch of the first image and/or with the patch of a second image. In other words, an alignment 130 of a portion of an image is performed based on the one or more local transformations determined for the respective one or more patches of said portion of the image.
20220031-02-039062-00155 [0084] Using the one or more of local transformations to align the image portions may reduce the complexity, especially in case the local transformation are pre-stored. Moreover, it may result in a more accurate alignment especially in views with high magnification. [0085] When referring to a patch or a region, what is meant is, for example, a part of a tissue scan. The patch may be defined as a combination of its location, size, and magnification level (resolution). The term “patch” is different from the term “FOV” or “view”, as it not necessarily related to those parts of the image that are being viewed or accessed. Moreover, the FOV or view may include one or more patches. [0086] For example, patches of a view may correspond to so-called tiles, i.e. a set of non-overlapping rectangular (square or non-square) patches spanning the tissue scan. Such tiles may form a regular set such as a grid. Tissue scans are often stored, accessed and processed as an image-pyramid comprising tiles of a fixed size at multiple magnification (resolution) levels. [0087] Fig.2 shows schematically an image 200, a first view 210 and a second view 220. The first view 210 includes patches 220 that may overlap and that are, within the view 210, defined by their location and size. The view 210 itself is specified by its location and size within the image 200. It may be further magnified (zoomed-in or out) when displaying on the screen so that it may be further characterized by the level of magnification (or resolution). The second view 250 includes patches 260 organized as tiles in a grid. The image 200 may be, for instance, the first image (moving image) or the second image (fixed image). [0088] The above described method may be implemented by a device 300 exemplified in Fig. 3. The device 300 comprises a processing circuitry 310. The processing circuitry 310 may itself comprise one or more processors and/or programmable hardware (such as one or more FPGAs) and/or application specific hardware (such as ASICs) and/or other pieces of electronics. The processing circuitry 310 may be configured to perform the method described above with reference to Fig.1 or further specific methods described below. The configuration of the processing circuitry 310 may be performed by programming the one or more processors to perform the method or its parts, programming the programmable hardware correspondingly, and/or designing the electronics elements and interconnections of the processing circuitry correspondingly. The processing circuitry may be connected, e.g. via a bus 390 with
20220031-02-039062-00155 further modules such as a storage module 320, a display control module 330, a communication interface module 340, and/or an operation input interface module 350. [0089] The storage module 320 may store the image 200 and/or may store the local transformations pre-calculated for one or more patches 220, 260 of the image 200 – possibly at different magnifications. For example, in some implementations, the image 200 may be stored in the storage module 320 in a form of an image pyramid, including the image 200 at different resolutions (number of samples per image). The local transformations may be stored for patches within each of the resolutions. However, this is only a non-limiting example which may further reduce computational complexity involved in accessing different fields of view and magnifications of the image and aligning them. In general, image 200 may be stored without such pyramidal structure, in its resolution, alongside the corresponding local transformations. The storage module 320 may be a volatile memory or a non-volatile memory or disc or the like. It is noted that, while shown herein as being a part of the device 300, the storage module may be an external storage and not necessarily part of the device 300. [0090] The display control module 330 is a circuitry that controls a display to display the image 200 or its portions before and/or after the alignment. The communication interface module 340 is a circuitry that facilitates communication with an external device. It may be configured, for example, to receive the image 200 from an image capturing device. The operation input interface module 350 is a circuitry configured for receiving commands instructing the device 300 e.g. to perform alignment, to select a specific view, or the like. [0091] A functional structure of the processing circuitry 310 is further exemplified in Fig. 4. Accordingly, the processing circuitry 310 includes a local transformation selection module 430 and an alignment module 440. [0092] For example, said obtaining 110 of the one or more local transformations is performed by selecting the one or more local transformations out of a set of local transformations. This may be performed by the local transformation selection module 430. As mentioned above, the set of local transformations may be stored in the storage module 320. Said selecting of the one or more local transformations may include retrieving the one or more local transformations from the storage module 320. The selection of pre-stored local transformation may reduce computational effort when repeatedly accessing the image or its views and aligning them or using them for alignment of other images.
20220031-02-039062-00155 [0093] Nevertheless, the local transformations are to be determined or calculated before they can be stored in the storage module 320. This determination of the local transformations may be performed in a local transformation calculation module 420 as will be exemplified in detail below. It is conceivable that the obtaining 110 includes the determination of the local transformations, e.g. by calculating them. The local transformations may thus be obtained on the fly and not pre-stored and merely selected. In such case, the local transformation selection module 430 would not be supplemented, but rather replaced by the local transformation calculation module 420. In this case, still the calculation of the local transformations may be less demanding on processing power than calculating a global alignment, and may be more precise than calculating global alignment in low resolution to safe the processing power. The calculation of the local transformation may be fully automated or aided by a human user. [0094] In order to increase accuracy of performing the local transformation calculation in module 420, a pre-processing module 410 may perform one or more processing steps on the image, as will be described below in more detail. [0095] The application of the local transformations for alignment in steps 120 and 130 may be performed by an alignment module 440, part of the processing circuitry 310. [0096] Use of local transformation(s) for alignment [0097] The term “transformed image” refers to an output of an alignment process. In particular, the transformed image may be obtained by transforming the moving image with the one or more transformations. [0098] Alignment 130 (by the transformation determined in step 120) of an image or its portion may be global or piecewise. The alignment is referred to as a global alignment, if a transformed image (or image portion) is obtained by applying a single transformation. The alignment is referred to as a piecewise alignment, if a transformed image (or image portion) is obtained by applying a plurality of transformations. The piecewise alignment may include locally aligning a view using a set of precomputed local (e.g. linear) transformations, each determined (e.g. optimized) for a respective region (e.g. patch). The regions are located at different locations within the tissue scan. Automation of the alignment process may be applied to consecutive sections, sequentially stained images and across stain modalities or the like. [0099] Fig.5 is a schematic drawing illustrating an image 500 with a scanned tissue. It further shows a view 550 of the image 500, located within the coordinates x and y of
20220031-02-039062-00155 the image 500 and having a size of sx in the horizontal direction x and a size of sy in the vertical direction y of image 500. [0100] In an exemplary implementation, each transformation ^^^ (i being an index from the range 1 to S) out of the set of S local transformations ^^^ to ^^ௌ is stored in association with a location ^^^ within the first image 500 and/or the second image for which said local transformation has been derived. The location ^^^ in the image 500 may be indicated, e.g. as the coordinates (within the image 500) of a top left corner 552 of the view 550 or as a center point 555 of the view. Dots such as dots 510 denote locations within the image 500 at which local transformations are available, for example included in the set of local transformations within the storage module 320. The location may be given in any unit such as samples of the image 500 or metric units or an index of a keypoint, block, tile, patch, any other image unit. [0101] Fig. 6 shows an image 600 (which in this case is similar as image 500) and multiple (S, in this example S=10) anchor points (locations ^^^) for which respective local transformations ^^^ are available (e.g. precomputed and stored in the storage module 320). In Fig.6, the labels of the location and transformation is included in a frame together with the respective point indicating the location for easier reference. Moreover, it shows a view 650 at a location L*. For example, the view 650 is being observed by a human user who desires to align this view with a corresponding view of another image. Such other image may be, e.g., a differently stained or captured under a different illumination or the like. In order to perform the alignment, thus, transformation ^^∗ has to be found. [0102] This may be performed by selecting 110 a plurality K of the S=10 local transformations. Herein, K is larger than 1 and smaller than or equal to the number S of the local transformations in the set of local transformations. Then, the transformation for aligning the portion of the first image to the portion of the second image is determined 120 to be a function of the K local transformations. In particular, the transformation ^^∗ for the field of view ^^∗ may be defined as follows: ^^∗ = ^^ ^ { ^^^ , ^^^ ^ , ^^ ∗ ) = ^^ ^ ( ^^^, ^^^), ( ^^ଶ, ^^ଶ), ( ^^ଷ, ^^ଷ), … , ^^ ∗) [0103] Function f can depend on transformations ^^^ , locations ^^^, and the location L* of the field of view 650. The Function f may be implemented as any interpolation function such as an average, a weighted average, nearest neighbor, or any other interpolation function. In an exemplary implementations, said function is the weighted
20220031-02-039062-00155 average and the weights are determined according to a distance between the location associated with the respective transformation ^^^ and the location L* of the portion (e.g. view 650) of the first image and/or the portion of the second image. For example, the weights may be inverse proportional or at least negatively correlated with the respective distances. [0104] After determining 120 the alignment transformation ^^∗ the transformation ^^∗ may be applied to transform (align) 130 the field of view 650. [0105] Such alignment approach may be seen as piecewise alignment, since for different views, the transformation ^^∗ may be recalculated, so that the transformations may differ for different views (pieces) of the image 600. It is noted that such piecewise alignment may be preceded by some pre-processing which may be seen as a rough alignment, so that the transformation ^^∗ may be seen as a fine alignment transformation ^^∗. [0106] It is conceivable to use (select) all S available local transformations. However, it may be advantageous for complexity and/or accuracy reasons to select a subset of K local transformations out of the S local transformations. [0107] According to an exemplary implementation, said selecting 110 the one or more local transformations includes selecting each of the one or more local transformations ^^^ among the S local transformations to satisfy a locality condition. The locality condition requires that a location ^^^ associated with said local transformation is in a distance ^^^ relative to the location L* of the portion 650 of the first image (or the second image) smaller than a neighborhood threshold ^^௧. The location L* may be, as mentioned above, for example a center or a corner or another reference point of the view. This approach may be referred to as a piecewise local alignment. [0108] Fig.7 illustrates such local alignment. In particular, an image 700 showing the same tissue as images 600 and 500 includes locations ^^^, ^^ଶ, ^^ସ, ^^ହ, and ^^^ associated with respective locations ^^^, ^^ଶ, ^^ସ, ^^ହ, and ^^^. The locations ^^^, ^^ଶ, ^^ସ, ^^ହ, and ^^^ are at respective distances locations ^^^, ^^ଶ, ^^ସ, ^^ହ, and ^^^ from the location L* of the view. Among these locations, only locations ^^^, ^^ହ, and ^^^ fulfill the locality condition. In particular, distances ^^ଶ, ^^ସ of the locations ^^ଶ and ^^ସ are larger than the neighborhood threshold ^^௧. Circle 770 encompasses all locations within the image 700, of which the local transformations may be used to determine 120 the alignment transformation ^^∗.
20220031-02-039062-00155 [0109] It is noted that the above mentioned locality condition based on the neighborhood threshold ^^௧ may be used alone or in combination with further conditions or rules. For example, the locality condition may be used to preselect a subset of P local transformation out of the S available local transformations. Then, a further selection mechanism may be applied to the set of P local transformations. Alternatively, another selection mechanism may be applied and then the selected local transformations may be further limited to those P local transformations that fulfill the locality condition. [0110] An example of a selection of a subset of local transformations is application of a K local neighbors. This example of selection may be used alone or together with other conditions or rules (e.g. the locality condition as mentioned above). In particular, said selecting 110 the one or more local transformations is selecting of K local transformations ( ^^^, with i being 1 to K) of which the associated location ^^^ is closest to the portion of the image (first and/or second image). K is larger than 1 and smaller than or equal to the number of the local transformations in the set of local transformations. [0111] The transformation ^^∗ for aligning the portion of the first image to the portion of the second image may then be determined to be a function (e.g. the function f mentioned above) of the K local transformations. [0112] For example, when K=2, in Fig.7 only the first two locations ^^^, ^^ହ closest to the location L* of the view would be selected. Then, ^^∗ =
^^ହ). If average is the function f, then ^^∗ = 0.5 ∗ ( ^^^ + ^^ହ). [0113] In general, in averaging K transformations at locations nearest to a view location L* as discussed above, the transformation is calculated as follows: ^^∗ = ^^ ^^ ^^ ^^ ^^ ^^ ^^ ( { ^^^ | ^^ ^^ ^^ ^^ ( ^^^ ) ≤ ^^ ^ ) [0114] Where ^^^ = ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^( ^^∗, ^^^) and rank is determined by sorting ^^^ from smallest ( ^^ ^^ ^^ ^^=1) to largest. If K= 5 , then ^^∗ = ^^ ^^ ^^ ^^ ^^ ^^ ^^( ^^^, ^^ଶ, ^^ସ, ^^ହ, ^^^) – first five closest neighbors are averaged. In some implementations the above-mentioned threshold distance ^^௧ may be added to the definition of the function: ^^∗ = ^^ ^^ ^^ ^^ ^^ ^^ ^^({ ^^^ | ^^ ^^ ^^ ^^( ^^^) ≤ ^^, ^^^ < ^^௧^) [0115] As discussed above, adding ^^௧ will result in ^^∗ = ^^ ^^ ^^ ^^ ^^ ^^ ^^( ^^^, ^^ହ, ^^^).
20220031-02-039062-00155 [0116] The above mentioned examples are not to limit the present disclosure. In that examples, the alignment transformation has been found, common for all samples of the view 650. Such transformation may be efficient especially in cases in which the misalignment is (or is assumed to be) homogenous for all samples within the view. [0117] If no such assumption is desirable, a subset of local transformations may be individually selected for pieces of the view 650. This is illustrated, for example in Fig. 8. An image 800 with a tissue similar to images 500, 600, and 700 includes a view that has two parts 850 and 860. The first part 850 of the view has a location ^^∗^, and the alignment transformation ^^∗^ for this part 850 is determined based on a first subset 810 of the local transformations. The first subset includes the local transformations ^^^ and ^^^. The second part 860 of the view has a location ^^∗ଶ, and the alignment transformation ^^∗ଶ for this part 860 is determined based on a second subset 820 of the local transformations. The second subset includes the local transformations ^^ସ and ^^ହ. It is noted that the first subset and the second subset are not necessarily disjoint, i.e. one particular local transformation may belong to both subsets 810 and 820. [0118] In general, the view can be divided to a plurality of parts, for each of which the alignment transformation is determined independently. In particular, for each such part, the set of local transformations is selected individually and then the alignment transformation is determined (computed) based on the thus selected local transformations. The actual selection of the local transformations may be performed as discussed above based on locality condition or K closest neighbors or the like. However, the present disclosure is not limited thereto and the selection may also take into account (for the entire FOV or for the parts) also the direction or homogeneity of alignment specified by the local transformations. [0119] In an exemplary implementation, a pixel-wise nearest K neighbors rule is applied. In particular, said selecting 110 the one or more local transformations is selecting of K local transformations individually for each sample out of a set of samples included in the portion of the first image. K is larger than 1 and smaller than or equal to the number of the local transformations in the set of local transformations. A transformation for aligning said sample of the portion of the first image to a sample of the portion of the second image is determined 120 to be a function of the K local transformations.
20220031-02-039062-00155 [0120] For instance, said function is a weighted average and the weights are determined according to a distance between the location associated with the respective transformation and the location of said sample. However, the present disclosure is not limited to this function and any function, for example the above mentioned function f, may be applied. [0121] Herein, sample is an image sample corresponding to a pixel and having one or more color components. The K local transformations may be the K local transformations nearest to the sample or the like. It is noted that the individual determination does not necessarily mean that the alignment transformation must be unique per pixel, but it means that it is determined for each sample separately and, as a result, may but does not have to differ from sample to sample. [0122] The set of samples mentioned above may include all samples of a view, but does not have to include all samples of the view. In particular, the selection 110 and the determination 120 may be performed for each second or third or fourth or the like sample in vertical and/or horizontal direction of the view in order to reduce complexity. Then, the remaining samples may be aligned by an alignment transformation determined for the nearest neighboring sample. For example, transformations for aligning respective samples belonging to the portion of the first image but not included in the set of samples are determined as a function (e.g. average, weighted average, copying from a closest neighbor in certain sample scanning order, or the like) of a one or more transformation determined for respective one or more samples in the set of samples. [0123] In the above mentioned examples, the alignment transformation could be determined 120 based on (e.g. as a function of) a plurality of selected 110 local transformations. However, the present disclosure is not limited to such approaches. According to an exemplary implementation, the transformation for aligning the portion of the first image to the portion of the second image is determined (selected) 120 to be one local transformation out of the set of local transformations. [0124] For example, a nearest local transformation is selected as is illustrated in Fig. 9. Image 900 includes a view 950 on a position L*. The transformation ^^∗ of the view 950 is determined to be the local transformation ^^ହ since this local transformation is associated with the location ^^ହ (of a point 910 in Fig. 9) that is closest (among the locations ^^^ to ^^^^ of available local transformations) to the location L* of the view 950.
20220031-02-039062-00155 [0125] In other words, when looking to method of Fig.1, said selecting 110 the one or more local transformations is selecting of one local transformation ^^୧ of which the associated location ^^୧ is closest to the portion 950 of the first image (or to the portion of the second image). The transformation ^^∗ for aligning the portion 950 of the first image to the portion of the second image is determined to be said one local transformation ^^୧. However, the present disclosure is not limited to the one local transformation being the closest transformation. Additional or alternative criteria may be used. [0126] Computation of the local transformations [0127] In the method described with reference to Fig.1 it was assumed that the local transformations are obtained in some way, e.g. calculated on the fly or read from the storage. In the following, some exemplary approaches on how to compute the local transformations belonging to the set of local transformations are described. [0128] In particular, as illustrated in Figs. 6 to 9, the local transformations ^^^ are associated with respective locations ^^^ for which they have been determined. These locations may be referred to as anchor points and may be specified in various ways. For example, they may be specified as key points or points of interest within the first image and the second image. Key points or points of interest are locations within an image that include a certain feature. Features are defined by a feature descriptor which is a rule or a formula that enabled determination whether or not a portion of image includes the feature. A feature may be, for instance, a presence of an edge or slope of the edge, or presence of a corner, or some color-based feature or the like, as is known to a skilled person. [0129] In particular, determining the set of local transformations is illustrated in Fig.10. In step 1010, key points are detected in the first image. In step 1020, key points (points of interest) are detected in the second image. For example, such points of interest may be determined as key points ( locations) in which one or more features are present (feature detection). Presence of the same one or more features are detected within the first image and within the second image. It is noted that the sequence of steps 1010 and 1020 may be reversed. However, the present disclosure is not limited to such key points or points of interest. [0130] In general, there does not need to be two separate or independent steps 1010 and 1020 of determining the key points in the first image and in the second image. For
20220031-02-039062-00155 example, the first image and the second image may be pre-aligned by means of a global transformation and brought into a common coordinate system. Then local transformations may be calculated based on selection of locations (key points, points of interest) in the common coordinate grid. This means that when finding local alignments, the points of interest may be determined only in the common coordinate frame, which may be the fixed image coordinate frame (second image). The term “pre- aligned” here refers to an alignment performed previously, e.g. before the currently described action. In the example above, “pre-aligned” refers to an alignment preceding the determination of the local transformations. [0131] The method of Fig. 10 further comprises obtaining 1030 a set of pairs of matching key points in the first image and the second image. Here, a pair of matching key points is a pair of key points considered to be located in the same location of the imaged tissue in both the first image and the second image, even though the location of the key points in the image coordinates may differ (e.g. due to misalignment of the images or parts of the images). For example, each key point is associated with a feature descriptor value. The matching is performed by evaluating similarity between the feature descriptor values, according to any of methods known in the art. Alternatively, for instance in a case of a global pre-alignment, the matching keypoints may be considered as co-located within a common coordinate frame. [0132] In step 1040, for each pair out of the set of pairs of matching key points (points of interest), the following is performed: [0133] - determining a local transformation for aligning a first patch located at the key point of said pair in the first image with a second patch located at the key point of said pair in the second image, and [0134] - including the local transformation into the set of local transformations. [0135] As already discussed with reference to Fig.2, patches are portions of an image which are defined by their size and location. A patch is a region at an anchor point (here a key point) for which the respective local transformation is found. “At the anchor point” means that there is a predefined spatial relation between the anchor point and the patch. For example, as shown the patch may be defined with an anchor point in the middle or with an anchor point in the top left corner or with an anchor point in another predefined location. The size of a patch may be in principle smaller than the view as illustrated in Fig.2. This may reduce the computational effort necessary to find the local transformation and may enable higher accuracy in case local misalignments
20220031-02-039062-00155 are expected. However, the present disclosure is not limited thereto and the patch may have the same or larger size. [0136] According to an exemplary implementation, the pairs in the set of pairs of the matching key points (points of interest) are obtained in the first image in a first resolution and/or in the second image in the first resolution. The first resolution may be lower than the full resolution, in order to reduce the computational effort. However, the present disclosure is not limited thereto. It is also possible to perform the matching in the full resolution, or even to perform the matching in more than one resolutions for the same images in order to increase robustness of the key point (point of interest) detection. [0137] Moreover, said determining 1040 a local transformation comprises extracting the first patch located at the key point (point of interest) of said pair in the first image in a second resolution, and extracting the second patch located at the key point (point of interest) of said pair in the second image in the same second resolution. In order to obtain local transformations with high precision, the second resolution may be higher than the first resolution. The second resolution may be the full resolution of the first image and the second image. [0138] Said determining 1040 a local transformation further comprises determining pairs of patch key points matching in the first patch and in the second patch; and deriving said local transformation according to the pairs of patch key points as a Euclidean transformation. Euclidean transformation is a transformation that includes only translation and rotation, but no zoom (scaling) or reflection or the like. Adopting Euclidean transformation only provides for a simpler implementation and a more robust matching. Moreover, it may be particularly suitable for the misalignments occurring in different scans of the same tissue. In other words, given a set of key points in the patch of the first image (first patch key points) and a set of respective matching points in the patch of the second image (second patch key points), the local transformation is determined as a translation and a rotation which best maps the first patch key points onto the second patch key points. The quality of the mapping may be determined by a cost function that may correspond to a distance metric between the location of the second patch key points and the location of the transformed first patch key points. Such distance metric may be a sum of distances or a sum of squared distances or the like.
20220031-02-039062-00155 [0139] The matching of the corresponding patches of the pair of the matching key points (points of interest) may be performed by detecting patch key points within the patches and matching them to obtain pairs of matched patch key points. This approach may be particularly suitable for the cases in which the first resolution was lower than the second resolution. [0140] The present disclosure is not limited to such patch matching for determining 1040 the local transformation. In general, it may not be necessary to extract patches from both images. It is conceivable to extract a patch corresponding to a first-image key point of a matching pair only from the first image and to then perform the local transformation detection by matching the extracted patch with the second image in the portion at the second-image key point of the matching pair. The matching may be performed by calculating a similarity metric between the extracted patch of the first image and multiple positions within the second image similar to motion vector estimation applied in video coding. In general, any of well-known image matching approached may be applied. [0141] Regarding step 1030, not all pairs of the matched key points (points of interest) that are found must be used as locations ^^^ and have a local transformation attached with them. In some implementations, it may be desirable to select only a subset of the pairs. For example, many key points may be located at the edges or in certain regions rich of texture. It may be sufficient for the purpose of alignment and efficient in terms of computational complexity to limit the amount of the pairs and thus the local transformations available. [0142] Correspondingly, Fig.11 shows a flow chart of an exemplary implementation which selects the pairs (of points of interest) in a more distributed manner. In particular, the determination 1030 of a set of the pairs of the matching key points in the first image and the second image comprises obtaining 1110 a superset of pairs of matching key points and selecting 1120, among the pairs of matching key points of the superset, the best matching pair of key points. Then, the best matching pair of key points may be moved from the superset to the actual set of pairs of matched key points that is then used to find the respective local transformations. The best matching pair may be determined, for instance, by a similarity metric such as a difference, absolute difference, square root of difference, correlation, or the like. [0143] Then the determination step 1030 may include repeating a plurality of times the steps of selecting 1130, among the pairs of matching key points of the superset, the
20220031-02-039062-00155 next best matching pair of key points that fulfills a predefined condition; and 1150 moving the next best matching pair of key points from the superset to said actual set. Repeating a plurality of times may mean repeating until all key points from the superset are checked or until key points of certain locations are checked, or a predefined number of times, or the like. [0144] The testing of the condition is shown in step 1140. As can be seen, in the figure, if the condition is satisfied, (“yes”), the moving step 1150 is performed, else the moving step 1150 is not performed and the algorithm continues with selecting 1130 the next best pair. The predefined condition includes a condition according to which the next best matching pair of key points is to be located farther than a threshold distance from any of pairs of key points moved to said actual set so far. In step 1160 it is tested whether all pairs from the superset have already been selected in an iteration of step 1130. If affirmative, the algorithm terminates. If negative, the algorithm continues with selecting 1130 the next best pair. The threshold distance may be fixed or configurable. [0145] In order to test the condition, it may be sufficient to test the location of only one key point the pair. For example, only the key point of the first image may be compared with the threshold distance. Alternatively, only the key point of the second image may be compared with the threshold distance. Still alternatively, an average of the key points of the pair may be compared with the threshold distance or both key points may be compared or the like. [0146] The flow chart of Fig.11 is only an illustration. In general, the selection of pairs in a distributed manner so that the selected pairs fulfill the above condition may be implemented differently. For example, rather than the processing pair by pair, the superset may be ordered from the best matching pair to the worst matching pair. Then, pairs not fulfilling the condition are removed from the superset. Then, the pairs remaining in the superset are used as the actual set of pairs of matching points. Other alternatives are possible. [0147] In other words, in step 1030, the set may be determined to include only pairs that fulfil a certain condition. The condition may require that a distance between any two key points in the first image (or the second image) is not smaller than the threshold distance. [0148] Sometimes, large rotations may bring inaccuracies. Moreover, in some applications, rotations are not expected to substantial. Accordingly, in some implementations, large rotations are filtered out. In particular, in said determining the
20220031-02-039062-00155 set of local transformations, local transformations that include a rotation by more than a threshold angle are not included into the set. The threshold angle is an angle larger than 0°. For example, the threshold angle may be 5° or 10°. It is conceivable to limit the local transformation to a translation only. However, this may limit the accuracy of the alignment for some applications. [0149] In the above described examples, the locations of the key points (points of interest) were determined primarily according to their matching degree. Some approaches to distribute the key points within the image have been mentioned. However, for some applications, it may be advantageous to provide the key points on more regular locations. [0150] According to an exemplary implementation, key point (point of interest) determination is performed per image unit as is shown in Fig.16. For example, said obtaining 1030 a set of pairs of matching key points in the first image and in the second image comprises dividing at least a portion 1600 of the first and/or second image into a pattern of image units 1650, and determining the set of pairs of matching key points 1690 to include one key point per image unit. [0151] For example, the pattern of image units may be a grid of tiles or blocks or the like. Fig.16 shows a regular grid of tiles of the same size. However, the tiles (or blocks) do not necessarily have the same sizes. For instance, hierarchical (recursive) splitting may be applied depending on the amount of texture or information of the respective image units (smaller image units if there is more texture in the image and larger image units if the image is smooth). [0152] Said one key point (point of interest) per image unit may be a single one key point per image unit, i.e. the key point belonging to the best matching pair of key points in the image unit. In other words, for each image unit location in the first image and the second image, the best matching pair of key points within the image unit on said location is determined. However, the present disclosure is not limited to this example, and there may be more key points per image unit. For example, the amount of or a maximum number of the key points per image unit may be predefined. [0153] In order to further simplify, it is conceivable to define the location of each key point always on the same position within the corresponding image unit, e.g. in its center. Then, the locations within the image unit does not need to be stored. The local transformations are associated directly with the index of the image unit.
20220031-02-039062-00155 [0154] As mentioned above, the determination 110 of the local transformations is not necessarily based on key points or not only on key points. Intensity-based local transformations may be used in addition or alternatively. [0155] According to an exemplary implementation, said determining 1040 the set of local transformations includes determining pairs of matching portions of the first image in the second image based on a similarity of image intensities between the first image and in the second image. Such matching may be performed by way of extracting patches around some matching key points (points of interest as described above) and then determining the matching patches directly (without referring to the patch key points), in order to determine the local transformation. The ter, “directly” here means that the luminance or brightness or color space components or the like of the patch are directly matched. This may be performed in any of known ways used e.g. for motion estimation or the like. The patches used in such direct matching are advantageously high-resolution patches, possibly full resolution patches in order to achieve pixel accurate alignment. [0156] The matching may be also performed without explicit determining (1010-1030) key points (points of interest) by e.g. dividing image into tiles or blocks and matching them. Hereby, not all blocks or tiles are necessarily used for matching. The decision on which blocks are used in matching may be performed based on the degree of texture present in the blocks. [0157] Said determining 1040 the set of local transformations includes deriving, for at least one of the pairs of matching portions (of the first image and the second image), a local transformation for aligning portion of said at least one of the pairs in the first image to the matching portion of said at least one of the pairs in the second portion. Such determined local transformations may then be included 1050 into the set of local transformations. [0158] It is noted that in any of the above-mentioned approaches, there may be more local transformations determined than finally included into the set of available transformations (and stored with the image). In order to reduce the amount of stored data and reduce the processing costs when using them, the determined local transformations may be evaluated as to their similarity, and same transformations that are associated with mutually adjacent locations may be removed. [0159] In any of the above-mentioned approaches, the matching with or without key points may be performed e.g. in an RGB intensity image or in one of the color
20220031-02-039062-00155 components or in a subset of the color components or the like. The present disclosure is not limited to any particular color representation. [0160] Determining a global transformation based on local transformations [0161] In some applications, it may be advantageous to perform a global transformation rather than local transformations. Accordingly, said determining 120 a transformation for aligning the portion of the first image to the portion of the second image is determining of a global transformation for aligning the (entire) first image to the (entire) second image. It includes calculating a function of the local transformations in the set of local transformations. [0162] The function may be an average or any other interpolation function. It is noted that the global transformation is not necessarily performed by averaging previously calculated local transformations. It is conceivable to evaluate the transformations by a different filtering to determine the main direction (e.g. by median) of the alignment. [0163] User-assisted alignment [0164] There may be various scenarios in which alignment is desirable. For example, alignment of serial sections (e.g. consecutive sections of the same tissue block, two adjacent slices from cut from the same tissue block not included in the same image) may be desirable. In such cases, piecewise alignment mode may enable manual setting of the local alignment transformations for each region of interest (for instance view) by a human user (e.g. a pathologist), based on recognizable matching local structures as has been described above. The combination of automated and user- assisted alignment may streamline the process of working with multiple aligned regions of interest, since the local transformations may be made persistent (reused for further views) as will be discussed below. In serial sections, some smaller misalignments may be generally expected, as they may be inevitable, since not all visible structure is of the same cells in both sections. Some cells may be larger and may be identified on both sections, but some of them may not be, or will not be present in the same way (e.g. caused by different cuts through a 3D structure). These mismatches would typically be smaller for consecutive sections than for sections further apart. [0165] For aligning when large-scale deformations occurs, finding multiple local transformations and presenting them using Piecewise alignment allows for robust high-resolution alignment as described above. Locations for determining a local transformation can also be preselected (manually). This may be particularly beneficial
20220031-02-039062-00155 for annotation in predefined regions of interest and for ensuring the best local alignment e.g. for training/validation dataset used for training is obtained. For setting the ROI centers as local transformation locations may guarantee accurate alignment where it may be most needed. [0166] Fig. 12 illustrates the device 300 described already with reference to Fig. 3, now as a part of a system that is capable of interacting with a human user. In particular, the device 300 is connected over its display control interface to a display device 360. The display device 360 may be a standalone screen external, and connectable and disconnectable from the device 300 or it may be a display device permanently connected to and being a part of the device 300. Such display devise may be any kind of display such as an OLED, LCD or the like. [0167] Moreover, a user input device 380 may be connected via the operation input interface 350 of the device 300. The user input device 380 may be a keyboard including one or more keys such as a standard computer keyboard or the like or any kind or keyboard. Alternatively, or in addition, the input device 380 may include a touch screen or a mouse or another means for pointing a cursor on various position for instance within a graphical user interface displayed on the display device 360. The communication interface 340 of the device 300 may be configured to connect the device 300 with a network 395. As illustrated in Fig.12, an image capturing device 370 (such as a slide scanner or the like) that is a source of the first image and the second image may be also connected to the network 395, so that the device 300 may obtain the first image and the second image directly from the image capturing device 370. However, this way of obtaining the images is only exemplary. The images are not necessarily obtained directly from the image capturing device 370. They may be obtained from an external storage over the network 395. The communication interface 340 may include or be an USB interface, so that the images may be obtained e.g. from an USB storage device or the like. [0168] The images may be already stored and/or obtained together with the sect of local transformations. Alternatively or in addition, the local transformations may be determined and stored together with the image by the device 300 as described above. [0169] Fig. 13 shows a graphical user interface (GUI) window 1300 that may be displayed by the device 300 on the display 360. The display shows a control panel including some standard icons 1320 that may comprise some standard functions such as an icon for opening of a file (e.g. a file including the first image and/or the second
20220031-02-039062-00155 image), for saving a file (e.g. the aligned images or the first image), “saving as” icon for saving a file in a desired file format, or an icon for copying (or duplicating) some portion of an image or the like. Moreover, the control panel includes a button / icon (or another kind of user input area) 1310 that causes the device 300 to perform an alignment of a part of the first image to the part of the second image. As can be seen in Fig.13, the GUI window 1300 displays in this example the second-image portion 1330 on the left hand side and the first-image portion 1340 on the right hand side. After clicking or otherwise activating the button 1310, the first-image portion 1340 is automatically aligned to the second-image portion 1330 according to one of the methods described above (e.g. with reference to Fig.1). It is noted that the second- image portion 1330 and the first-image portion 1340 may already be pre-processed which may include a pre-alignment, e.g. by applying a global transformation or local transformations in a lower resolution or the like. [0170] According to an embodiment, a human user may assist in obtaining the local transformation. [0171] In an exemplary implementation, said obtaining 1030 a set of pairs of matching key points in the first image and the second image comprises receiving at least one or more of the a set of pairs of matching key points from a user. For example, the receiving the at least one out of the set of pairs of matching key points from a user includes displaying the portion (view) of the first image and the portion (view) of the second image so that the first image is pre-aligned with the second image (in the displayed part). [0172] This is shown in Fig.14. In particular, Fig.14 shows a GUI window 1400. The GUI window includes a first area for displaying the portion 1440 of the first image and a second area for displaying the portion 1430 of the second image. As can be seen, the portions 1430 and 1440 are pre-aligned. Although there is a slight misalignment, the portions include same parts of the tissue. Star marker 1410 in both portions 1430 and 1440 denotes center coordinates (anchor points or key points) of the pre- alignment. The pre-alignment may be a result of an automated rough and/or fine alignment as described in this disclosure or a result or some human or machine learning based pre-alignment. The present example with two images displayed side by side is not to limit the present disclosure. Rather, there may be more than two images and they are not necessarily displayed side by side, but may be displayed adjacent in vertical direction and/or in horizontal direction.
20220031-02-039062-00155 [0173] Moreover, the GUI window 1400, similarly to the GUI window 1300 of Fig.13, comprises in the control panel some usual icons 1320 and may (but does not have to in this view) comprise the button 1310 “Find the best local alignment automatically” described above. In addition, the control panel comprises areas (icons or buttons or the like) 1450 and 1460. The area 1450 is associated with function of user input and referred to in Fig. 14 as “Add local alignment control points pair”. The area 1460 is associated with function of registering local alignment based on the user input and referred to in Fig.14 as “Register local alignment”. [0174] Once the user input activates (e.g. clicks on) the area 1450, the graphical user interface (GUI) provides user with means for marking at least one position on the displayed first image and on the second image. For example, the user may mark the first pair of key points 1420, shown in the first area and in the second area as triangles and labeled by “1” in a frame. [0175] After marking of the pair 1420 of matching key points by the user, the device 300 (in particular the operation input interface module 350) detects the at least one position (location ^^^) within the displayed first image and second image marked using the GUI. Correspondingly, based on the detected at least one position, the device (e.g. the processing circuitry 310) determines and records the at least one out of the set of pairs of matching key points. [0176] Then the user may mark further matching pairs of key points, as is shown in Fig.14: a second pair is denoted by circles labeled with “2” in a frame and a third pair is denoted by squares labeled with “3” in a frame. [0177] In Fig. 14, if the user then activates the area 1460, the device 300 (e.g. the processing circuitry 310) stores the local alignment indicated by the user by way of marking the key point pairs. Area (e.g. a button) 1460 may perform two actions: 1. Calculating a new local transformation (e.g. upon new registration of the images) and 2. registering the local transformation to be part of local transformation list. However, these two actions may also have two dedicated areas (buttons) of the GUI associated with them and operable by a user. Providing a single button simplifies the interface. Providing separate buttons may allow the user to check the calculated transformation and decide whether it is to be actually registered. [0178] For example, the key point (point of interest) location ^^^ may be stored, alone or alongside with a local transformation ^^^ that may be computed based on the marked
20220031-02-039062-00155 key point (point of interest) pair. Alternatively, the marked pair(s) or key points may be used to determine or correct the local alignment transformation associated with the pre-alignment center point 1410. [0179] Fig.15 illustrates an example of a GUI window 1500 that provides a user with additional or alternative input opportunity. In particular, a second GUI area on the left hand side displays the second image (in particular a view or the second image) 1530. A first GUI area on the right hand side displays the first image (in particular a view or the second image) 1540. The center 1510 of the pre-aligned views 1530 and 1540 is marked with a star. The user may be provided keys 1580 that may be physical keys (e.g. on a keyboard) or virtual keys shown within the GUI or both. The keys may have the function to move the first image (view) 1540 relative to the star 1520. Accordingly, the user may move the image until it is correctly aligned to the second image 1530. Fig.15 shows a button / icon 1550 in the control panel denoted as “Add local alignment using keys”. When this button / icon is activated (e.g. clicked on) by the user, the user may use the keys for the above-mentioned alignment functionality of moving the first image view 1540 to be aligned with the second image 1530. The resulting alignment may be then determined and recorded upon activating the icon / button 1460. The button / icon 1550 may be present alternatively or in addition to the button / icon 1450 of Fig.14. [0180] In general, the user does not have to determine or mark the pair of key points, but may directly input the local transformation. An example has been described with reference to Fig.15 with limitation to translation (using the usual up, down, left, and right keys). However, the present disclosure is not limited to such interface. [0181] For example, said determining the set of local transformations includes providing a user with a graphical user interface (GUI) allowing a user to: [0182] - display the first image (or first-image view which is a view of the first image) and/or the second image (or second-image view which is a view of the second image) in a selectable resolution, [0183] - apply at least one of desired rotation, translation, and scaling to the displayed first image and/or second image (wherein these operations correspond to applying of a local transformation), and [0184] - add a local transformation resulting from the at least one of desired rotation, translation, and scaling into the set of local transformations.
20220031-02-039062-00155 [0185] Selectable resolution means that the user may be allowed (e.g. by means of the GUI) to select the desired resolution of the first-image view and the second-image view. The applying of the local transformation can be performed by the user using any input device 380 such as a mouse a keyboard or a touch screen or the like. [0186] It is noted that in any of the examples mentioned in the present disclosure, the locations and the respective transformations associated with the locations may be stored together with the resolution or magnification for which they were determined. User-selected key points or user-selected local transformation as described above may help especially in cases the automated alignment does not work accurately enough and may be used to supplement any of the automated alignment approached discussed herein. [0187] The GUI may be designed based on UI or UX design. UI focuses on visual interface elements such as typography, colors, menu bars, and more, while UX focuses on the user and their journey through the product (user experience). Some possible UI and UX methods for adding the local transformations as discussed above in scenarios where the fixed and registered images to be viewed side by side (horizontally as shown in figures or vertically), such that the views are aligned according to the updated state of the local transformations. [0188] In particular, the GUI may provide input means (such as buttons, icons, touch areas, adjustable shapes such as lines / rectangles, cursors or the like) for selecting and marking a transformation center on the fixed image and for marking the matching point on the registered image (moving image). Such input means may be part of the control panel in the window shown in Figs.13-15 or 18-20 or accessible through it. [0189] It is noted that when referring to “input means” in connection with GUI, these are not limited to displayed areas accessible by mouse or arrows of a keyboard. Input means may also enable direct entry of coordinate by using a numerical keys or the like. Moreover, speech control may be used or the like. [0190] The marked points (marked by the user as mentioned above) are then used to calculate a transformation as also already discussed with reference to Fig. 14. The transformation may be for instance a translation transformation (consisting of linear movement). The order in which the points are marked is not necessarily important (at first the moving image then the fixed image or vice versa). The rotation part of the calculation can then be found (if desired) using feature-based or intensity-based alignment methods applied to a region surrounding the marked points. This region
20220031-02-039062-00155 may be a patch of a predetermined size, not necessarily the entire view or a tile or the like. [0191] In some exemplary implementations, the user can be provided input means for marking a second matching pair of points on the fixed image and moving (registered) image, which will be used (possibly in conjunction with the first marked pair) to calculate the rotational transformation. In other words, the user can mark two different pairs of anchor points, one used to obtain a translation transformation and another one to obtain rotation movement. The moving image may but does not have to be displayed as aligned according to the translation transformation before the user selects the second pair of anchor points. [0192] However, the user does not have to mark the two pairs of key points sequentially, it is possible to mark them and then to let both transformations (translation, rotation) be calculated. It is noted that the present disclosure is not limited to marking two pairs of key points and that a user may be provided means (e.g. GUI means as described above) to mark more than two pairs and then to determine an optimal transformation considering all marked pairs. [0193] The present disclosure is not limited to providing user a possibility to mark points. The GUI may be configured to provide a user an alternative or additional means to marking a line on the fixed image and marking a matching line on the registered (aligned) moving image. The local transformation is then calculated e.g. using the two paired ends of the respective lines. In general, either the origin or the terminus or both of the line can be taken as the location for the new local transformation. [0194] In some GUI implementations, when points or lines are marked on either image, a corresponding point or line may be automatically added to the other image. The user then only needs to move the points or lines to the correct (matching) locations. This may make the interaction of the user with the viewer easier and more intuitive. [0195] As another option, marked points or lines may be moved (dragged) by the user using a pointer control such as a computer mouse or the like, triggering a recalculation of the transformation, until a sufficiently accurate transformation is obtained. The triggering may be automatic (once movement event or release of the pointer device is detected, optionally time-throttled), or manual, e.g. by clicking a button, a keyboard key or a similar UI element. [0196] Another alternative or additional option is, instead of moving points using a pointer device, to provide a control device specifically for rotation, such as a physical
20220031-02-039062-00155 button or dial, or a GUI element corresponding in appearance to such rotatable button or dial and to provide a control device specifically suitable for translation (which may be physical arrow keys or a graphical representation of the same functionality on the GUI). [0197] In another alternative or additional option that may facilitate user interaction, one or more points or lines are marked on the fixed image only. the control device for rotation and/or for translation is used to rotate and/or move the registered image until the image features visually match the pattern of points/lines marked on the fixed image (similarly to arrow keys in Fig.15). A GUI comprising one or more of buttons, icons, inputs fields, or dials may be used instead or in addition to a physical control device. The physical control device may be a keyboard with keys and possibly a dial or a panel on a viewing device including such keys and/or dial(s). [0198] Similarly to Fig. 15, it is not necessary to provide a user with a possibility to enter points or lines. The user may use either the physical control device for rotation and/or translation or a corresponding GUI comprised of buttons/inputs/dials to manually move and rotate the moving image within the viewing area until a sufficiently accurate alignment is achieved. The location for the new transformation in such cases may be predefined, such as the center of the view, which may be intuitive to the user. [0199] The transformation for the views may be calculated using either feature-based or intensity-based methods described above. For every change of view (such as panning), a local transformation aligning the current views may be automatically calculated using either feature-based or intensity-based methods described above. It is noted that such a local transformation may be applied after and in addition to a prior (preceding) alignment (pre-alignment) that may use local transformations calculated and stored as described above. [0200] Optionally, the calculation is time-throttled. Optionally, the calculation can be distance threshold, such that a new transformation is calculated only when panning has been performed by the user beyond a certain pre-set distance. Optionally, the resultant calculation may be saved (cached or permanently stored with the image) such that it would not be recalculated when viewing the same view again. [0201] Pre-processing and post-processing [0202] Fine alignment described above using high-resolution views of the scanned images may start from a roughly aligned pair of images (the first image and the second image referred to above) and is meant to improve on rough alignment for example by
20220031-02-039062-00155 using higher-resolution features than those used for a rough alignment. The rough alignment may be obtained in an automated manner within a pre-processing or by a human for instance using a graphical user interface or the like. [0203] A multi-stage alignment may thus be provided as illustrated in Fig.17. Fig.17 shows the followings steps: [0204] Step 1710: Scale, reference frame, and/or image type reconciliation [0205] Step 1720: Identifying main tissue part(s) in the scanned images [0206] Step 1730: Rough alignment using a low-resolution view of the scanned images [0207] Step 1740: Fine alignment using high-resolution views of the scanned images [0208] Step 1750: Pixel-perfect alignment using an elastic alignment (such as Optical Flow) [0209] It is noted that step 1740 comprises the fine alignment including obtaining 110 said one or more local transformations and determining 120 the alignment transformation accordingly. [0210] Step 1710, according to an exemplary implementation, comprises, before obtaining 110 said one or more local transformations: [0211] - scaling of the first image to match a ratio between a unit of length and an image sample size in at least one direction; the at least one direction being vertical and/or horizontal direction, and/or [0212] - transforming the first image including flipping the first image around a vertical axis and/or flipping the first image around a horizontal axis. This may be performed to reconcile the reference frame for both the first image and the second image, and/or [0213] - converting the first image and/or the second image into a predetermined image type. [0214] In particular, for a meaningful alignment of images, they should to be presented and accessed using the same reference frame and scale definitions. Moreover, the images may be desired to use the same image format (image type). However, this is not necessary. In general, images from different scanners which are saved using different types may be also aligned. Moreover, for example, it is possible to align RGB images and single-channel fluorescent images which are inherently not of the same type. In other words, extracting of features may still be performed on images with different types and then, the matching is still possible. For example the feature extraction may be performed by a machine learning model (such as a neural network),
20220031-02-039062-00155 one trained to extract features from fluorescent images and a different one to extract features from RGB images, or the like. [0215] The scale may specify, for instance, the ratio of microns (micrometers) to a pixel (sample). In general, the scale specifies the ratio between real world measure of the captured tissue and a number of image samples of an image representing the tissue. One possible representation of the scale is a magnification level of an image pyramid. An image may be available in different magnification levels in which case the scale may be given by the magnification level. For example, a level 0 may be defined as 40 times magnification of the real-world object (also denoted as X40), level 1 may be defined as 20 times magnification of the real-world object (also denoted as X20) and so on. [0216] The scaling includes modifying the scale of the first image to match the scale of the second image. Alternatively, the scaling may modifying the scale of the second image to match the scale of the first image. In general, it is of no importance which of the images is used as a reference. The aim is to obtain the first image and the second image having the same scale, e.g. having the same microns to pixel ratio, irrespectively of how this is achieved. Thus, it is conceivable to scale both the first image and the second image to a scale different from the scales of the first and the second image. [0217] In some cases, one of the images obtained from the image capturing device may be scanned in a “mirrored” reference frame (e.g. left/right or up/down reflection). This may usually be a systematic effect dependent on the definition of a slide scanner or scanning configuration. In such cases, before the alignment, it may be necessary to transform all images (that are to be aligned) to the same reference frame, since the alignment transformations comprising translation and rotation (possibly zoom) may not properly correct for reflection. This is what was above referred to as flipping the image around the vertical axis and/or horizontal axis. However, implementations are possible that find transformations which include translation, rotation and possibly (e.g. optionally) a reflection. Supporting reflection alongside with translation and/or rotation may further improve automation e.g. in cases in which the slide is placed in a scanner upside down or the like. [0218] Regarding image type reconciliation, the images to be aligned may be in a different image type, such as brightfield or grayscale or a multi-spectral image. In other words, the color space or in general image settings may be different. In such cases, a
20220031-02-039062-00155 reconciliation of the image type (settings) may be desirable. For example, fluorescence may be in some applications considered as grayscale with a single channel. However, it is also possible to provide a composition of multiple fluorescence channels (e.g. taken using different light sources such as LEDs/filters). The image settings are not necessarily only color space settings. Rather, the image settings may include pixel representation. For example, pixels of the RGB are in the format 3Xuint8 (three times unsigned integer represented each by 8 bits, i.e. one pixel is represented by 3 color components red, green, and blue, each component represented by an 8-bit unsigned integer). On the other hand, grayscale may be represented by floating point precision, or by an (unsigned) 16-bit integer representation. Alignment might, although not necessarily, require normalizing to some specific numerical representation. [0219] It is noted that step 1710 may also include a step of determining whether the first image and the second image are in the same scale, reference frame, and/or image type. This may be determined for instance by referring to meta data stored with or within the image file. Alternatively or in addition, it may be determined automatically (e.g. by a corresponding software module). [0220] Step 1710 may further comprise a step of providing a user a choice to perform scale change, reference frame change or image type change manually. Step 1710 may provide the automatic reconciliation of the scale, reference frame and/or the image type as another choice for the user. The choices may be presented to a user via a graphical user interface that may be displayed by the display controller 130. [0221] It is noted that step 1710 may be omitted for example in case there is no reason to assume that mismatch in scape, reference frame or image type can occur. In other words, the present disclosure does not require step 1710 as a part the above described fine alignment. [0222] Step 1720, according to an exemplary implementation, comprises, before obtaining said one or more local transformations a step of distinguishing, in the first image and/or in the second image, a foreground from a background. Then step 1720 may further include detecting, within the foreground, a largest connected region. For example, the largest connected region may be assumed to correspond to the main tissue parts. The distinction of the foreground and background may be performed according to any known approaches. [0223] According to an exemplary implementation, the distinguishing the foreground from the background is performed by an entropy-threshold-based method which
20220031-02-039062-00155 outputs a bitmap indicating, for each unit of the first image, whether said unit belongs either to the background or to the foreground. Such unit may be a sample of the image (also referred to as “pixel”). The first image and/or the second image on which the foreground / background detection is to be performed may be subsampled so that the detection is performed on a subsampled (lower-resolution than the resolution of the first image and/or the second image) image. This corresponds to detecting whether an image unit belongs to a background or to a foreground for units larger than an image sample. Such units may be blocks or tiles (e.g. rectangular or square) or other image regions. [0224] The actual tissue frequently fills only a part of the scanned image, since it does not have to cover the whole slide that has been scanned to obtain the first image and/or the second image. The rest of the image is a background signal, which does not typically contribute to the alignment as it tends to be homogeneous. Moreover, oftentimes slides may include additional non-tissue marks, such as scratches and pen markings, which should be ignored when performing the alignment. To overcome these issues, the above mentioned custom entropy-threshold-based method for identifying foreground image-pyramid tiles at full magnification level or lower. If the entropy threshold based method is applied to the full resolution (highest magnification), a foreground sample mask (binary mask) may be obtained for each sample of the full resolution, corresponding, as mentioned above, to foreground tile mask (binary mask) in lower resolutions. The identifying of the connected regions of the foreground tile mask or the foreground sample mask may be performed according to any well-known approach. The recovered regions may then be filtered using a maximum Solidity threshold, finally selecting the largest of the remaining foreground regions as the main alignment region. [0225] In order to further reduce the computational complexity, it may be desirable to perform the foreground/background detection and/or the identification of the main alignment region in a third resolution that is lower than the resolution of the first image and/or the second image. [0226] The main alignment region is then used for a rough alignment in step 1730. In other words, step 1730 includes performing a rough alignment of the first image to the second image by roughly aligning the largest connected region detected in the first image to the second image or to the largest connected region detected in the second image. For completeness it is noted that alignment of the second image to the first
20220031-02-039062-00155 image is also possible. In order to further reduce the computational complexity, it may be desirable to perform the rough alignment in a resolution lower than the full resolution of the first image and/or of the second image. This may but does not have to be the same as the third resolution mentioned above. The third resolution may be a predetermined resolution, such as a resolution lower than a threshold. Third resolution may be same as the first resolution, but does not have to be, it can be even lower than the first resolution. [0227] The rough alignment of the largest connected region to the second image comprises, for example: [0228] - detecting, in the foreground of the first image (possibly also in the foreground of the second image to increase robustness) in said third resolution, key points and feature descriptors associated respectively with the key points; [0229] - detecting, in the second image in said third resolution, key points and feature descriptors associated respectively with the key points; [0230] - obtaining rough-resolution pairs of key points by matching the key points and the feature descriptors detected in the first image with the key points and the feature descriptors detected in the second image; [0231] - selecting a subset of the rough-resolution pairs of key points with the best matching; and [0232] - deriving a rough transformation for said rough alignment according on the subset of the pairs of key points. [0233] In order to maintain efficiency, the rough alignment is performed in the third resolution that is lower than the full resolution and also lower than resolutions applied after the rough transformation for further alignment stage or stages. [0234] In one advantageous implementation, the key points may be detected only in the largest-connected foreground region of the first image and the second image. This is when step 1720 of distinguishing the main connected region is applied. However, the present disclosure is not limited to such approach and the rough transformation may be applied to all regions of the image, so that step 1720 is not necessarily performed. Moreover, it may be possible to identify the main connected region in one of the images (e.g. the first image) only and looking for the match in the entire other image (e.g the second image). Step 1720 is independent of step 1710. In particular, step 1720 may be employed irrespectively of whether or not step 1710 was performed. It is further noted that at any of the stages, additional filtering or processing may be
20220031-02-039062-00155 applied such as white balance or color space conversion or the like. In addition or alternatively, color unmixing or deconvolution in order, for example, to perform alignment based on nuclei positions only (hematoxylin) may be performed. It is noted that such processing could be performed before the fine alignment. For example, it could be performed between steps 1730 and 1740, in particular if the fine alignment 1740 required some modifications that are not necessary or advantageous for the rough alignment 1730. [0235] The above mentioned selecting a subset of the rough-resolution pairs of key points with the best matching may be, for instance, selecting a predefined amount (e.g. N) best matching pairs or pairs of which the match (measured by some similarity measure) exceeds certain threshold – similarly as described for the matching of key points applied in the local-transformation based approach described above. The rough transformation may be a Euclidean transformation. [0236] Deriving the rough transformation may be a determining of the transformation by a Random sample consensus (RANSAC). RANSAC is a known iterative approach for estimation of model parameters (in this case the rough alignment transformation) from a set of observed data (here the key points) that contains outliers, when outliers are to be accorded no influence on the values of the estimates. RANSAC distinguishes between inliers and outliers (here key point pairs that do not fit well into the transformation model). Thus, it may be particularly suitable for the rough transformation in which more outliers can be expected than in case of local transformation determination. However, RANSAC may also be applied in the stage of the fine alignment. [0237] Rough alignment using low resolution view of the scanned images may help to save processing power, and, in combination with the fine alignment, achieve a desired accuracy. [0238] In a specific example, the main tissue regions of the two images are extracted 1720 at a prescribed low-resolution magnification level (in parties, this could be, for instance an X5 (five times) magnification), and white balanced (this may comprise excluding exact white and exact black values). From each of the two low-resolution images, a number of key points (locations) and associated feature descriptors are extracted. For this purpose, Oriented FAST and Rotated BRIEF (ORB) features from the OpenCV library but other feature detectors and descriptors can be used instead.
20220031-02-039062-00155 [0239] Then, matches are determined between the feature descriptors from the two images, keeping the best top several matched pairs for computation of the alignment. The matched key points are used to find a (e.g. Euclidean) transformation (e.g. comprised of rotations and translations only), using a robust RANSAC implementation. In principle, affine transformation may be used in some application, which includes scale and shearing terms. However, Euclidean transformation may be advantageous for tissues with nuclei and cells, in which the addition of scaling and shearing terms may in some cases result in some distortions, especially when aligning consecutive sections. [0240] Post-processing [0241] For a very fine alignment on pixel (image sampling or sample) scale, after applying said transformation for aligning the portion of the first image to the portion of the second image, an additional alignment e.g. by optical flow is applied to the first image. It is noted that optical flow is only one of possible postprocessing options. In general, any elastic or non-rigid alignment may be applied. [0242] Optical flow is a representation of relative motion between two images for one observation point. It is typically represented by a raster of locations within an image and motion vectors (in general, local transformation in case the motion is modelled to include more than a linear movement as was described above) associated with each location and denoting movement necessary to come from the second image to the first image (or vice versa). In the present disclosure, the optical flow may be determined for each sample of the first image and denotes local transformation necessary to come from the sample of the first image to a corresponding sample of the second image. Any approach known in the art for determining optical flow may be applied. With such optical flow obtained, an alignment may be performed on the precision of samples. As is known to those skilled in the art, sub-sample approaches are also possible, based on inter sample interpolation. [0243] As a specific mere implementation example, once the sequentially (or fluorescently) stained images are aligned to within a few pixels or a few tens of pixels by the fine alignment, an optical flow method can be applied to achieve pixel-perfect alignment. For example, an OpenCV DIS Optical Flow algorithm may achieve a good alignment with fast running times. However, any other algorithm may be employed depending on the desired functionality and application.
20220031-02-039062-00155 [0244] In some cases, the calculated Optical Flow field calculated for a pair of images may be applied to additional images which are pre-aligned with the transformed image. For example, both fixed and moving (transformed) images may be brightfield scans, and the moving image may be naturally aligned to a fluorescence scan of the same tissue taken in parallel to the brightfield scan. In this case, the Optical Flow field calculated for the pair of brightfield images may also be applied to the fluorescence scan. Correspondingly, the methods and devices presented herein may be configured to store calculated optical flow together with the images. [0245] In another exemplary use case, the Optical Flow may be calculated between DAPI fluorescent scan depicting cell nuclei and a brightfield Hematoxylin scan also depicting cell nuclei. The calculated flow field can then be applied to additional fluorescent channels scanned in parallel to the DAPI channel. [0246] The application to additional moving images in the above mentioned use cases is not limited to optical scan and may also apply to local transformations. [0247] Further modifications and specific examples [0248] In the above examples, cases were described, in which two images were aligned. However, all presented examples can be extended to align more than one pair of images. For example, one image among the plurality of images to be aligned is defined as the fixed image (second image) and the remaining images are defined as moving images (as described for the first image above). The alignment transformations is then determined for each moving image to align it with the one fixed image, which remains unchanged. [0249] Alternative solutions are also possible in which a first image is a fixed image for a second (moving) image, but the second image after alignment is a fixed image for a third (moving) image, etc. Especially when aligning more than two images, sometimes it may be easier to align certain pairs of images, since they have some overlap in the stains used or in the tissue features depicted by those stains. In such a case it may be preferable to find the alignment transformation using a pair of fixed and moving images and then applying the transformation to a third image, which is pre-aligned to the moving image. This can be particularly useful when aligning multiple fluorescent images to a brightfield image, where one of the fluorescent scans depict cell nuclei and the brightfield image includes Hematoxylin stain which also stains the nuclei. [0250] Another situation where this is useful is where a grayscale brightfield image is taken using a single channel camera, which is also used for scanning fluorescent
20220031-02-039062-00155 images without moving the tissue slide. The grayscale image can be aligned to a stained brightfield image (e.g., H&E or some IHC) and the computed transformation can be applied to the fluorescent images which are naturally pre-aligned to the grayscale image. This form of applying a transformation found on a pair of images to a third image is applicable for all forms of alignment described above, Rough Alignment, Fine Alignment and Optical Flow (in general, an elastic transformation). [0251] In summary, the alignment of the present disclosure may have multiple stages such as a pre-alignment (rough alignment); alignment based on local transformations (fine) – may be local, for image portions, or global for entire images; and postprocessing. In these stages, a use of transformations is made. It is noted that the transformations for different stages may differ in what they model. For example, as mentioned above, Euclidean transformation may be particularly suitable for the local transformation based alignment. The optical flow models linear movement of each pixel. This is only an example. The selection of the transformation type is to be done with regard to the kind of tissues and the scanning process, because inter alia they have impact on the kind of expected misalignment. In some application, scaling may be applied as a part of a transformation at one or more stages. [0252] The herein described methods and apparatuses may be used in wide spectrum of applications, and may be particularly suitable for aligning biological or biochemical tissues. For example, the first image and the second image are images of the tissue, sequentially scanned and/or differently (for example sequentially) stained. [0253] A rough alignment that uses low resolution images (views of the scanned image) may but does not need to be applied before the fine alignment that uses high resolution images (views of the scanned images). When referring herein to a piecewise alignment, what is meant is the process of locally aligning a view using a set of precomputed (possibly linear) transformations, each optimized for a region at different locations along the tissue. As mentioned, in the above embodiments, aggregations of local transformations may be applied to the moving image. They may be manually defined by user or automated. An aggregation of local transformations may be performed using a variety of mathematical functions other than the averaging mentioned above. The present disclosure is not limited to the particular exemplary implementations presented above. The automatic alignment may be a part of the fine alignment, such as the piecewise local features-based alignment or piecewise local intensity-based alignment mentioned above.
20220031-02-039062-00155 [0254] Same transformation may be applied to the whole of the moving image (first image), as has been mentioned in an above section relates to global transformation, and in particular global feature-based alignment in which matching key points are determined in the images and the global alignment transformation is determined according to the determined key points. Another global transformation determination possibility is an averaged local features-based alignment in which the global transformation is obtained by averaging (or another function) of the local alignment transformations. [0255] Several approaches for the fine alignment, each with advantages for different types of pairs of images, have been described above: [0256] - Global features-based alignment [0257] - Averaged local features-based alignment [0258] - Piecewise local features-based alignment [0259] - Averaged local intensity-based alignment [0260] - Piecewise local intensity-based alignment [0261] All of these approaches may start by finding a set of matched key points (points of interest) at a low resolution (similarly as in rough alignment, e.g. X5). Unlike in the rough alignment during pre-processing an additional nearest neighbor condition may be applied when selecting the matched key points to use (possibly, the rough alignment has already been applied before the fine alignment started). [0262] With this condition selecting the best match may be performed, and then the next best match can be determined which is also at least some distance threshold far from the previously selected matches. In this way it is possible to proceed until enough matched key point pairs are selected, which are also spread across the tissue image. Without the additional condition many of the matched key points may tend to be clustered together, often on the edges of the tissue, as has been explained above. [0263] Alternatively, a set of matched key points may be provided to the algorithm. For example, by manually selecting roughly matched key points on the fixed and rough- aligned images, e.g. regions of interest selected by a pathologist for detailed analysis. Alternatively, a grid of roughly matched key points spanning the main tissue region is used. Whichever way the key points are selected, high-resolution patches centered on each key point are extracted from the fixed and moving images. It is noted that the grid may be formed by tiles. The patches are not necessarily as large as the tiles. The patches may be smaller than the tiles which may be faster, or larger than the tile size,
20220031-02-039062-00155 which may be more robust and result in less apparent discontinuities e.g. in case of persistent viewing / accessing of the images. [0264] The different fine alignment methods may differ in how these patches are used to find the fine transformation. [0265] In a global features-based alignment, for each pair of patches, key points and feature descriptors are extracted and matched. The best matches for each pair are collected. Patch-pairs for which no matching key points are found are disregarded. For example, a Euclidean transform is calculated for the joint set of matched key-points using a RANSAC optimization. This method returns a single global transformation to be used as a global fine alignment. It is very useful when aligning sequentially stained images where the two images share some of the same stains used (for example both images include Hematoxylin to stain cell nuclei). When referring here to “global” it is not necessarily (but may be) global for the entire image. It may be global for a view or a portion of the view (e.g. in case of persistency, the new part of the view).. [0266] In averaged local features-based alignment, as in the global feature-based alignment method, matched key-point are collected for each patch pair. However, the matched key points are not collected. Instead, a separate local (e.g. Euclidean) transform is found for each patch-pair, based on the set of matching key-points from that pair. The resulting local transformation matrices are combined (e.g. averaged) to yield a single global transformation. [0267] Patch pairs where no matching key points are found or where finding a transformation failed, are disregarded, and are not included into the combined global transformation. [0268] In piecewise local features-based alignment, local transformations are computed as in the averaged local feature-based alignment method, but the resultant local transformations are not combined. Instead, they are collected (e.g. together with their key point coordinate) and used to generate a piecewise fine aligned image, which is described below. Also here, and actually in any of the examples herein, patch pairs where no matching key points are found or where finding a transformation failed, may be disregarded. [0269] In averaged local intensity-based alignment, image patch pairs are collected as in the above described methods. Instead of finding matching features in each pair, however, an (any known) intensity-based method is used to find a local transformation. In some cases, spatial auto-correlation is applied as an initial step or any approach
20220031-02-039062-00155 known from block matching in video coding or pattern matching or the like in object detection. The local transformations are then combined (e.g. averaged) to yield a global fine alignment transform. [0270] In a piecewise local intensity-based alignment, local transformations are computed as in the averaged local Intensity-based alignment method, but the resulting local transformations are not combined. Instead, they are collected (e.g. together with their key point coordinate) and used to generate a piecewise fine aligned image, which is described below. Note that for all local alignment methods, patch pairs where the local transformation computed includes a large rotation may be discarded to further reduce distortions. This may be particularly advantageous in filtering out undetected bad matches (e.g., due to artifacts), especially if the moving image is already roughly aligned to the fixed image, in which case large rotations are no longer expected. [0271] When using any of the piecewise local alignment methods described above, the result is a set of local transformations. When viewing such images, the alignment transformation for the selected view is determined by the proximity to the pre- calculated local transformation. For example, the image viewer can select the nearest local transformation to the current view. Alternatively, the nearest K transformations are averaged using a weighted average based on the distance from the current view. Alternatively, some interpolation method (e.g., bilinear interpolation) is used to derive a local transformation from the precalculated set. [0272] Still, alternatively, the transformation for each pixel in the current view is calculated according to the K nearest or all of the local transformation, possibly weighted by proximity, to generate a flow field which can be applied to the registered image view. [0273] Alternatively, the flow field is calculated for some pixels in the view (e.g. in a grid pattern) based on the local transformations and the flow field is then interpolated for all pixels in the view. [0274] “Alternatively” herein does not mean that one device could not implement the alternatives. On the contrary, a device 300 such as those shown in Figs.3 and 12 may implement any one or more alternatives mentioned above, which may be selected automatically based on application and/or by an interaction with a user. It merely mean that these approaches may be used without each other.
20220031-02-039062-00155 [0275] The piecewise transformations (transformations based on local transformations) may facilitate exploration, annotation and model-training tasks and have the potential to be impactful for clinical use as well. [0276] As mentioned above, interactive refinement of transformed image may be applied. [0277] In a specific exemplary implementation, the general case may be applying a list of local piecewise transformations, each associated with a certain location on the tissue. Transformations computed by any of the methods described above yielding a single global transformation (global for the entire image or global for a view or global for a portion of the view) may be associated with the center of the main tissue region (this is true for both rough and fine alignment methods). The list of local transformations can be extended at any time (e.g. at different viewing sessions), by adding a new local transformation with an associated location. The addition of the new transformation can be done manually, or using a GUI allowing the user to select an interest point on the fixed or transformed image and running either a feature-based or intensity-based automatic local alignment for the patched-pair around the selected point. [0278] Alternatively, the user may use a GUI for applying rotation and translation to the transformed image in the current view, and when the desired alignment is achieved choose to save the current transformed state as a local transformation associated with the location of the current view. The GUI may provide a user with an opportunity to change the location of the view while preserving the transformation, i.e. to move the anchor point. [0279] In summary, a method for automated global and local registration (alignment) of digital pathology sequentially scanned (same tissue-section) and consecutive sections images. The method additionally allows for interactive incremental refinement of tissue alignment (automatic/manual) for regions of interest selected by a user (pathologist). Such multi-stage alignment process may yield global, local (piecewise) and possibly (e.g. for sequential staining) pixel-perfect alignment. Piecewise alignment enables interactive refinement of automated alignment, allowing a pathologist to select regions of interest for locally improved alignment where global alignment didn’t fit well. This incremental refinement process combines the ease of use of automated alignment and the detainment of user control over which parts of the tissue are important for analysis.
20220031-02-039062-00155 [0280] User interface for viewing images involving alignment [0281] The portion of the first image and/or the portion of the second image may be a field of view, FOV, of the first image and/or the second image. The FOV is selected by a user. For example, the selection may be performed via the operation input interface module 380. [0282] As mentioned above with reference to Figs.12 to 15, the user may be provided with a GUI. For example, the FOV is selectable (and consequently also selected) by using the GUI. However, the selection of the FOV is not necessarily performed by GUI, it may be performed via a text input or another kind of input. [0283] Moreover, the method for aligning the portion of a first image to the portion of a second image as described in any of the above examples (e.g. with reference to Fig. 1) may be applied for the purpose of viewing the first image and the second image by a human viewer such as a medical practitioner or the like. For that purpose the processing circuitry 310 of the device 300 may control the display control module 330 to display aligned images in various manners as will be discussed in the following examples. [0284] In some exemplary implementations, upon selection of a (new) FOV (e.g. per GUI or in another way), said determining the transformation for aligning the FOV of the first image to the FOV of the second image according to the one or more local transformations is automatically performed. Moreover, the FOV of the first image is aligned to the FOV of the second image. [0285] This is illustrated in Fig. 18. On the left hand side, a GUI window 1840 is illustrated. The window comprises a control panel possibly with some icons or buttons or the like. Moreover, the window comprises a first area for displaying a FOV of the second image and a second area for displaying a FOV of the first image. The first area in this illustration contains a first FOV (denoted as FOV1) 1810 that has been selected by the user and that is a portion of the second (fixed) image. Tiles marked with a star ( ) in Fig.18 are transformed according to the ^^∗(FOV1) computed for the (newly) selected FOV1. Same marker color/filling (in window 1840 it is black) indicates tiles transformed according to the same transformation. It is noted that the marking by stars in Fig.18 is only for visualization. The GUI does not have to show any such marking. In fact, the GUI does not have to show the tile boundaries at all. Although the marking by the star is shown in the fixed image, it is noted that the transformation is applied to
20220031-02-039062-00155 the corresponding moving image portions (after the transformation corresponding to the fixed image portion) as described above. [0286] In this example, the same transformation ^^∗(FOV1) is applied to all tiles spanning the FOV1. The second area in this illustration contains a FOV 1820 of the first (moving) image that has been aligned by using ^^∗(FOV1) to correspond to the FOV1 1810 of the second (fixed) image. [0287] A transformation ^^∗(FOV) of a FOV is computed every time a new FOV is viewed (or, in general, accessed). On the right hand side of Fig.18, a GUI window 1890 is shown, with a control panel same as in window 1840. The window 1890 also comprises a first area for displaying a FOV of the second image and a second area for displaying a FOV of the first image. The first area in this illustration contains a first FOV (denoted as FOV1) 1810 that has been selected by the user and that is a portion of the second (fixed) image. The first area in this illustration contains a second FOV (denoted as FOV2) that has been newly selected by the user and that corresponds to a new portion of the second (fixed) image. Tiles marked with a star ( ) are transformed according to the ^^∗(FOV2) computed for the newly selected FOV2. Again, the same marker color (in window 1890 it is white framed by black) indicates tiles transformed according to the same transformation. In this example, the same transformation ^^∗(FOV2) is applied to all tiles spanning the FOV2. However, transformation ^^∗(FOV2) is recalculated upon selection of FOV2 and may thus differ from the transformation ^^∗(FOV1) calculated for FOV1. The second area in this illustration contains a FOV 1860 of the first (moving) image that has been aligned by using ^^∗(FOV2) to correspond to the FOV11850 of the second (fixed) image. [0288] In summary, Fig.18 shows a dynamic adaption meaning that with each new FOV selection, the transformation for the new FOV is computed and alignment is performed. In the example of Fig. 18, said determining the transformation is determining of a transformation common to all samples of the new FOV. This is represented in this example by the FOV divided into tiles and all tiles being transformed by the same transformation (marked by the respective black stars). The determination 120 of the common transformation may be performed, as described above, e.g. by averaging the transformations over all tiles of the FOV or by selecting one among the local transformations associated with the FOV portions (here tiles) and use it for all tiles.
20220031-02-039062-00155 [0289] An example of a persistent (static) behavior upon selecting a new FOV is illustrated in Fig.19. Accordingly, said determining the transformation 120 includes a determining of a transformation common for a part of the new FOV that was not part of a preceding FOV. The preceding FOV may be an immediately preceding FOV or not immediately preceding FOV. In other words, in a static adaption, the transformations for each sample (or, in this example, tile) can be persisted beyond the immediately preceding FOV. [0290] Fig. 19 shows an example with a window 1840 on the left hand side and a window 1940 on the right hand side, similarly to Fig.18. Since the window on the left hand side is the same as described in Fig. 18, the repetition of the description is omitted here. In case a user that has viewed the first FOV (FOV1) pans or otherwise selects a new (next) FOV (FOV2), the window 1940 includes the second (fixed) image in the left window area. In Fig.19, the image area also illustrates which transformation is applied to each tile of moving image to correspond to the FOV2. In the right window area, the corresponding moving image portion aligned to FOV2 is illustrated. In particular, tiles marked with a star are transformed according to a transformation that is new to the respective FOV. Tiles marked with a dot are transformed according to a previously computed, persisted transformation. Same marker color indicates tiles transformed according to the same transformation. Thus, FOV1 in window 1840 is transformed with ^^∗(FOV1) – the same transformation for all tiles determined upon selection of the FOV1. [0291] On the other hand, after a panning or otherwise selecting the FOV2 in window 1940, tiles 1910 that have been already present in FOV1 are still transformed with ^^∗(FOV1). This is indicated by the black dots. However, for tiles 1920 that were not part of FOV1, a common transformation ^^∗(FOV2) is determined and applied. This is indicates by white, black framed stars. The moving image 1960 is then aligned accordingly, using two different transformations. Using two different transformation may introduce some discontinuity on the border 1980 between the tiles aligned with ^^∗(FOV1) and the tiles that are aligned with ^^∗(FOV2). On the other hand, the persistence allows for reduction of computational costs and for continuity of motion, e.g. when panning or otherwise moving the moving image. Moreover, in the persistency mode you are the same pixels are obtained each time the FOV is accessed.
20220031-02-039062-00155 [0292] Fig.20 illustrates what may happen in case, after selection of FOV2 as shown in Fig.19, another new, third FOV (FOV3) is selected. As in Fig.19, here also a FOV transformation is computed every time viewed/accessed. The new transformation is only applied to tiles which were not viewed/accessed previously. Tiles marked with a black star are transformed according to the new ^^∗(FOV3). Tiles marked with a dot (full or empty) are transformed according to previously computed, persisted ^^∗(FOV1) and ^^∗(FOV2). Same marker filling (full, empty or crossed) indicates tiles transformed according to the same transformation. It is noted that the ^^∗(FOV1), ^^∗(FOV2), and ^^∗(FOV3) are respective transformations determined for respective portion of the image and they may be, as a result, different from each other. However, depending on the actual misalignment of the image content, they may also be same. [0293] In Fig.19, a horizontal pan was performed, whereas in Fig.20 a vertical pan is performed. Correspondingly, a discontinuity 2050 may now occur in some cases on a border between each two of the three transformations. It is noted that discontinuity is not necessarily present: if the misalignment is substantially uniform within the FOV, the static/persistent approach may reduce computation effort while still providing an appropriate alignment. It also ensures consistency, such that the same pixel in the transformed image, once viewed, will subsequently be shown in the exact same way. [0294] The persistence duration may be one of the following three options: [0295] a) Only the transformation determined for the immediately preceding FOV persists. This means that in case of a new, third FOV selected, only the transformation calculated for the last (second) FOV persists. Samples (or tiles) which were present in FOV1, but not in FOV2 are handled as samples or tiles new to FOV3, i.e. a common transformation is determined for these samples and the new samples of FOV3. [0296] b) Persisting until the viewing/accessing of the aligned images is terminated. In other words, when the images are closed in the GUI, or the file containing the images is closed, or a model training using the images is finished, then any persistency is lost. After opening the same file / image again, a new transformation common to the entire first viewed FOV is calculated. [0297] c) Persisting indefinitely, in which case the sample or tile specific transformations are stored into storage alongside the images for future viewing/access.
20220031-02-039062-00155 [0298] Option a) may be advantageous as a tradeoff between the viewing quality and computational power. Option b) may provide benefits in terms of computational complexity and be sufficient smooth for some applications. Option c) may be the fastest because it stores previous local alignments. With more viewings, the accuracy may be improved. [0299] Another possible option is persisting as long as tiles are (even in part) in the current view. This will ensure continuity while panning but will not persist when moving away from a view and returning back later. Another option is to manually switch between dynamic and static modes, in which case persistence may be cleared when switching to dynamic mode. [0300] As assumed in Figs.18-20, the GUI may provide for displaying the first image and/or second image. The selection of the new FOV may be performed by moving the displayed first image and/or second image including panning, zooming in/out, and/or moving to a pre-selected coordinate, or the like. For some applications, the first image is not necessarily shown alongside the second image. For example, the fixed image may still be accessible in the background, but is not shown as part of the GUI. It could be that the user (or any other process) will use only the first image (moving). Transformation may still need to be applied and the transformed image is the output. For example, annotations may have been marked on the fixed image (regions, points indicating locations of nuclei etc.) and the alignment allows looking at the transformed image in the context of these annotations. [0301] It is noted that the above static and dynamic access/view modes may both be supported by a device such as 300. In particular, two modes of rigid transformation application may be supported, the static and the dynamic. [0302] In the dynamic mode, each time a FOV is requested (e.g. by panning, zooming or by an API call or the like) a transformation is computed (or selected) from the list of the precomputed local transformations, and applied to the entire FOV. This mode prevents any discontinuities in the FOV due to mismatching local transformation. Optionally, the FOV transformation may be further tuned manually (e.g. as described above with reference to Figs.13 to 15) or in an algorithmic manner (e.g. as described above feature-based, intensity based, optical flow). Optionally, the fine-tuned transformation may be added to the set of local transformations, to be used for subsequent FOVs. Optionally, optical flow methods may be applied to the rigidly- transformed FOV.
20220031-02-039062-00155 [0303] The static mode is similar to the dynamic mode, but the FOV transformation is persisted for all tiles spanning the FOV, such that when a different, yet overlapping, FOV is requested, the persisted transformation is applied to the overlapping tiles, and a new transformation is computed for the non-overlapping tiles only. The new transformation may be then persisted for the non-overlapping tiles when further FOVs are selected. This mode ensures that the same tiles will always undergo the same transformation, but may lead to discontinuities between tiles undergoing different transformations. [0304] A GUI may be provided to allow the user for recalculation of FOV transformation for a FOV (e.g. current FOV or via API), which is then persisted for all tiles spanning the triggered FOV (this is a semi-dynamic mode). In other words, the user may be allowed to select various modes of persistence and/or to request recalculation of the transformation for the current view which would then reset the persistence and provide a common transformation for the entire view. [0305] Optionally, the transformation for each tile may be obtained (from a storage or computed) regardless of the FOV, in which case it may be precomputed for each tile, or computed on the fly when tiles are requested. [0306] The switching between the static and dynamic modes may be manual (button, toggle) or automatic based on the FOV parameters (e.g. magnification level, location or size). For example, for a pathologist viewing the images, the dynamic mode may be more desirable. When applying any kind of tile-based algorithm on the image it may be desirable to use the persistent mode to ensure consistency. [0307] Fig.21 schematically illustrates obtaining of consecutive slides. A breast tissue block 2100 is sliced into slices (sections) 2150 which include two consecutive slices respectively posed on slides 2151 and 2152. In this example the consecutive sections on slides 2151 and 2152 are sequentially stained with HER2 GE001 IHC and with HER2 qIHC. Fig.21 shows a specific kind of tissue and staining. However, in general, any kind of tissue and a suitable staining approaches may be used. [0308] Fig.22 illustrates piecewise aligned images 2201 and 2202 of HER2 IHC and qIHC stained consecutive slides, such as the slides 2151 and 2152 of Fig.21. Dots to which arrows 2280 are pointing indicate locations of respective transformations T1 and T2. These transformations may be determined as described in the embodiments above. For instance, the transformation T1 and/or T2 may be determined as the local transformation in the location indicated by the dots or as a transformation obtained
20220031-02-039062-00155 based on a plurality of local transformations. Discontinuity lines 2241 and 2242 illustrate separation of each slide image into two portions (upper and lower). Upper and lower portion of a slide image 2201 (and/or 2202) are aligned independently of each other, the upper part using transformation T1 and the lower part using transformation T2. The discontinuity lines 2241 and 2242 thus indicate where the piecewise transformed image is discontinuous. The discontinuity lines 2241 and 2242 may be also explicitly indicated in the respective aligned images 2201 and 2202 when displayed; or it may be absent in the displayed representation (e.g. in a GUI such as those exemplified above). [0309] Such piecewise alignment may provide advantages such as a more accurate alignment of local structures in the image. As shown in Fig.22, two tumor regions 2261 and 2263 of relevance in a tissue (annotated with a polygon forming a contour of the respective tumor region) are far from each other, and the piecewise alignment can thus allow for an accurate alignment locally using the transformations T1 and T2. Similarly, two tumor regions 2262 and 2264 in the second slide 2202 may be located in portions of the slide 2202 that are aligned independently from each other (possibly causing a discontinuity, illustrated by the discontinuity line 2242. [0310] In an exemplary implementation, as mentioned above, some regions may be beforehand annotated as regions of interest. For example, parts of an image may be classified, i.e. associated with some desired classification (e.g. tumor/non-tumor, tumor grade etc.). The tumor regions 2261-2264 in Fig.22 may be an example of such region of interest detection. Then, the piecewise alignment may be used to provide a good local alignment of the transformed image to the annotated regions and/or to a fixed image in the regions of interest. [0311] Any of the above mentioned methods may be implemented in a form of a program. For example, a computer program is provided, stored on a non-transitory medium and comprising code instructions which, when executed on one or more processors perform the steps of any of the methods. [0312] Moreover, devices (apparatuses) are provided that are capable of executing the above mentioned methods. For example, an apparatus 300 such as the one illustrated in Fig.3 is provided for aligning a portion of a first image of a tissue with a portion of a second image of the tissue. The apparatus 300 comprises processing circuitry 310 configured to obtain one or more local transformations based on the portion of the first image and/or the portion of the second image. Each of the one or
20220031-02-039062-00155 more local transformations is for aligning a patch of the first image to a patch of the second image matching the patch of the first image. Each of the one or more local transformations is associated with the patch of the first image and/or with the patch of a second image. The processing circuitry 310 is further configured to determine a transformation for aligning the portion of the first image to the portion of the second image according to the one or more local transformations. [0313] The apparatus may further comprise: [0314] - a storage device 320 storing the one or more local transformations, [0315] - an input module 350 configured to obtain the portion of the first image and/or the portion of the second image; and [0316] - an output module 330 configured to output images including the portion of the first image and/or the portion of the second image. [0317] The processing circuitry 310 is configured to obtain one or more local transformations from the storage device. [0318] The input module 350 may include at least one key and/or a touch screen and/or a voice input device; and the output module 330 may include a screen and/or the touch screen. [0319] The portion of the first image and/or the portion of the second image is a field of view, FOV, of the first image and/or the second image. The input module 350 may be configured to enable a user to select the FOV. In particular, the input module 350 and the output module 330 may be at least in part implemented by a graphical user interface, GUI. The “at least in part” refers to the possibility that there still may be voice control or keyboard or output into storage or the like. [0320] According to an exemplary implementation, the processing circuitry 310 is configured to, upon selection of a new FOV by a user over the input module, automatically perform said determining the transformation for aligning the FOV of the first image to the FOV of the second image according to the one or more local transformations, and to the possibility that there still may be voice control or keyboard or output into storage or the like. The processing circuitry 310 is further configured to obtain an aligned FOV of the first image by aligning the FOV of the first image to the FOV of the second image. The output module 330 is configured to output the aligned FOV. [0321] In an exemplary implementation, the output module 330 is configured to display the first image and/or second image, and/or the input module 350 is configured to (e.g.
20220031-02-039062-00155 by processing circuitry or another controller.) receive an input from the user for said selecting of the new FOV by moving the displayed first image and/or second image including panning, zooming in/out, and/or moving to a pre-selected coordinate. [0322] The GUI may implement further input and output features as described above with reference to Figs.13-15 and 18-20. [0323] Training AI [0324] As mentioned above, the present disclosure may be applied to provide a better viewing of related tissue images by a human user. Moreover, some implementations mentioned above provide for an efficient human machine interaction during viewing, e.g. by applying a certain degree of persistence and allowing the users to manually adjust the locations for which transformations are to be computed and/or to manually adjust the transformation itself. Such approaches may be suitable for assisting human users to annotate tissue image data e.g. for use in training machine learning models such as neural networks and neural network based pathology recognition tools or the like. [0325] For example, a machine learning (ML) model may be trained on parts of the image, which are beforehand annotated as regions of interest. For example, parts of an image may be classified, i.e. associated with some desired classification (e.g. tumor/non-tumor, tumor grade etc.). Then, a piecewise alignment may be used to provide a good local alignment of the transformed image to the annotated regions and/or to a fixed image in the regions of interest, without introducing non-Euclidean distortions to biological morphology. [0326] For example, a static-mode piecewise alignment can assure consistency in the data that the model sees during training. Moreover, a precomputed static-mode piecewise alignment can be used to save computations during an already computation-heavy model training and ensure consistency between different model training runs using the same data. [0327] Even without annotations, when using one image to project information on the second image, it is desirable to be as accurate as possible with the alignment. This may not be possible when using single global alignment on the whole image. These methods of projecting info from one slide to the other allow to generate massive ground-truth for AI training. [0328] In other words, a method may be provided for training a machine learning model, the method comprising aligning the images to be used for training to each other
20220031-02-039062-00155 and then entering the result of the alignment (aligned images) to the training of the machine learning model. [0329] While certain features and aspects have been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and/or any combination thereof. Further, while various methods and processes described herein may be described with respect to particular structural and/or functional components for ease of description, methods provided by various embodiments are not limited to any particular structural and/or functional architecture but instead can be implemented on any suitable hardware, firmware and/or software configuration. Similarly, while certain functionality is ascribed to certain system components, unless the context dictates otherwise, this functionality can be distributed among various other system components in accordance with the several embodiments. [0330] Moreover, while the procedures of the methods and processes described herein are described in a particular order for ease of description, unless the context dictates otherwise, various procedures may be reordered, added, and/or omitted in accordance with various embodiments. Moreover, the procedures described with respect to one method or process may be incorporated within other described methods or processes; likewise, system components described according to a particular structural architecture and/or with respect to one system may be organized in alternative structural architectures and/or incorporated within other described systems. Hence, while various embodiments are described with or without certain features for ease of description and to illustrate exemplary aspects of those embodiments, the various components and/or features described herein with respect to a particular embodiment can be substituted, added and/or subtracted from among other described embodiments, unless the context dictates otherwise. Consequently, although several exemplary embodiments are described above, it will be appreciated that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims. [0331] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be
20220031-02-039062-00155 apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. [0332] It is expected that during the life of a patent maturing from this application many relevant machine learning models will be developed and the scope of the term machine learning model is intended to include all such new technologies a priori. [0333] As used herein the term “about” refers to +/- 10 %. [0334] The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to". This term encompasses the terms "consisting of" and "consisting essentially of". [0335] The phrase "consisting essentially of" means that the composition or method may include additional ingredients and/or steps, but only if the additional ingredients and/or steps do not materially alter the basic and novel characteristics of the claimed composition or method. [0336] As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof. [0337] The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features from other embodiments. [0338] The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment of the disclosure may include a plurality of “optional” features unless such features conflict. [0339] Throughout this application, various embodiments of this disclosure may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the disclosure. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed
20220031-02-039062-00155 subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range. [0340] Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween. [0341] It is appreciated that certain features of the disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the disclosure, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the disclosure. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements. [0342] Although the disclosure has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims. [0343] It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present disclosure. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is/are hereby incorporated herein by reference in its/their entirety.