WO2021240545A1

WO2021240545A1 - A method and system for whole slide imaging with low computational complexity

Info

Publication number: WO2021240545A1
Application number: PCT/IN2021/050513
Authority: WO
Inventors: Adarsh Natarajan; Harinarayanan Kuruthikadavath Kurussithodi; Abhay Kumar
Original assignee: Adarsh Natarajan
Priority date: 2020-05-26
Filing date: 2021-05-26
Publication date: 2021-12-02

Abstract

Disclosed herein is a method and system for creating a pyramid of whole slide images of a sample. The system acquires plurality of source images from a glass slide stained with test sample and computes transformation required for the acquired source images. The system determines dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile. The system generates one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the tiles maps to one or more source images. The system applies transformation to each of the one or more source images identified based on determined dependencies for each tile and generates WSI for a level of the pyramid without requiring large memory for processing the whole slide images.

Description

Title: “A METHOD AND SYSTEM FOR WHOLE SLIDE IMAGING WITH LOW

COMPUTATIONAL COMPLEXITY”

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority to Indian Provisional Patent Application Number 202041022047, filed on May 26, 2020, the entire contents of which are hereby incorporated by reference.

FIELD OF THE INVENTION

Embodiments of the present disclosure are related, in general to whole slide imaging, and more particularly, but not exclusively to a method and system for whole slide imaging with low computational complexity.

BACKGROUND

Whole slide imaging refers to scanning of conventional glass slides in order to produce digital whole slide images (WSI)s. Pathology departments implement the whole slide imaging as the most recent imaging technique for enabling tele-medicine, automated diagnostics, slide archival, collaboration etc. Pathologists replace the act of viewing glass slides from microscope to digital display by using whole slide imaging system, wherein the pathologists can navigate the digital slide in the similar manner they navigate a map in computer, or mobile application. The digital whole slide image further enables the pathologist to scale or descale the image as required. Healthcare professionals and researchers can use the digital whole slide images for plurality of diagnosis of diseases and research works.

Conventional methods of whole slide imaging system acquire small images that span the entire slides and combine the acquired images to create a whole slide image that is large in size, wherein such processing requires huge memory. The whole slide images are stored in tiled pyramidal format to avoid the challenges related to viewing such images. The image stored in a tiled pyramidal format has plurality of scales stored as independently accessible components, wherein each scale is divided into smaller images or tiles. However, in the process of tiled pyramidal format creation, the acquired images are first transformed and a whole slide image (WSI) is created from the transformed images, wherein large size uncompressed pixel values are stored in the memory. Consequently, the creation of WSI becomes unfeasible on computing devices with moderate memory. Therefore, it is desirous to have a method and system that creates a tiled pyramidal WSI without the requirement of large memory and avoids the need to have devices or systems with high computing power.

The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgement or any form of suggestion that this information forms prior art already known to a person skilled in the art.

SUMMARY

One or more shortcomings of the prior art are overcome, and additional advantages are provided through the present disclosure. Additional features and advantages are realized through the techniques of the present disclosure. Other embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed disclosure.

Accordingly, the present disclosure relates to a method for creating a pyramid of whole slide images of a sample. The method includes receiving a set of source images of the sample acquired from a glass slide stained with the sample. The method comprises computing transformation required for each of the source images to accommodate in a Whole Slide Image (WSI). The method further comprises generating a WSI of each level in the pyramid. The method of generating the WSI for each level in the pyramid comprises determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile. The method further comprises generating one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images. The method includes applying transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

Further, the disclosure relates to a system for creating a pyramid of whole slide images of a sample. The system comprises an imaging device, a processor, and a memory communicably coupled with the processor. The processor is configured to receive a set of source images of the sample acquired by the imaging device from a glass slide stained with the sample. The processor is configured to compute transformation required for each of the source images in order to accommodate the source images in a Whole Slide Image (WSI). The processor further generates a WSI for each level in the pyramid that comprises a base level and at least one higher level. The processor is configured to generate the WSI for each level by performing steps of determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile. The processor generates one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images. The processor further applies transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

Furthermore, the present disclosure relates to a non-transitory computer readable medium including instructions stored thereon that when processed by at least one processor cause a system to receive a set of source images of the sample acquired by the imaging device from a glass slide stained with the sample. Further, the instructions cause the processor to compute transformation required for each of the source images in order to accommodate the source images in a Whole Slide Image (WSI). Furthermore, the instructions cause the processor to generate a WSI for each level in the pyramid that comprises a base level and at least one higher level. Further, the instructions cause the processor to generate the WSI for each level by determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile; generating one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images. Further, the instructions cause the processor to apply transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

The foregoing summary is illustrative only and is not intended to be in anyway limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. In the figures, the left- most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of device or system and/or methods in accordance with embodiments of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:

Figure 1 illustrates an exemplary architecture of a proposed system to create a whole slide image in a tiled pyramidal format in accordance with some embodiments of the present disclosure;

Figure 2 illustrates an exemplary block diagram of a whole slide imaging system in accordance with an embodiment of the present disclosure.

Figure 3A-3B illustrates an exemplary representation of pyramidal format of a whole slide image stored in different levels in accordance with an embodiment of the present disclosure;

Figure 4 illustrates a flowchart showing a method for creating a pyramid of whole slide images in accordance with some embodiments of the present disclosure; and

Figure 5 illustrates a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure.

The figures depict embodiments of the disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.

DETAILED DESCRIPTION

In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternative falling within the spirit and the scope of the disclosure.

The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a device or system or apparatus proceeded by “comprises... a” does not, without more constraints, preclude the existence of other elements or additional elements in the device or system or apparatus.

Embodiments of the present disclosure relates to a method and system for creating a pyramid of whole slide images (WSI) of a sample. In one embodiment, the system creates the WSI in a tiled pyramidal format by splitting the WSI creation process into small units of independent work that create individual image in the pyramid. An imaging device acquires a plurality of images of predetermined size from a glass slide to capture whole slide area of sample stained on the glass slide. The system receives the plurality of acquired images and determines size of whole slide image based of the position of the acquired images. Further, the system computes transformation required for each of the source images in order to accommodate the source images in a Whole Slide Image (WSI). Upon computing the required transformation, the system generates a WSI for each level in the pyramid, wherein the pyramid comprises a base level and one or more higher levels.

In order to generate the WSI for each level, the system determines dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile. The system generates one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images. The dependencies are determined by computing overlapping source images for each of the one or more tiles in the WSI. The system applies the one or more transformations to the identified overlapping images, blends all the transformed images and saves relevant area as part of whole slide image, wherein the transformed images contribute to formation of base level of whole slide image pyramid. The system further recursively creates the higher levels of the pyramid based on the images of the base level by manipulating the image size and following the analogous approach of creating the base level of whole slide image pyramid. The ability of the system to create the WSI pyramid directly from the acquired images without the need of storing the WSI in memory as a single file avoids the bottlenecks on scaling and memory.

In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

Figure 1 illustrates an exemplary architecture of a proposed system to create a whole slide image in a tiled pyramidal format in accordance with some embodiments of the present disclosure.

As shown in Figure 1, the exemplary system (100) comprises one or more components configured for enabling whole slide image creation in tiled pyramidal format from images of sample acquired from scanning of glass slides containing the sample. In one embodiment, the system (100) comprises a whole slide imaging system (WSIS) (102), an imaging device (104), and an enterprise database (106) communicatively coupled via a communication network (110). The communication network (110) may include, without limitation, a direct interconnection, LAN (local area network), WAN (wide area network), wireless network, point-to-point network, or another configuration. One of the most common types of network in current use is a TCP/IP (Transfer Control Protocol and Internet Protocol) network for communication between database client and database server. Other common Internet protocols used for such communication include HTTPS, FTP, AFS, and WAP and other secure communication protocols etc.

The imaging device (104) may be an electronic microscope, microscope-based slide scanner, high resolution image scanner etc., including the functionality for communicating over the communication network (110). The imaging device (104) enables the users for example, an authorized user of an enterprise or organization to interact with the WSIS (102) to acquire a plurality of source images from a glass slide stained with test sample. The imaging device (104) enables authorized owners of WSIS (102) to transmit the acquired source images directly to telepathology centres, wherein the telepathology centres perform plurality of operations on the digital format of the glass slide for remote diagnosis. For example, the imaging device can be a conventional electronic microscope, scanner, high resolution camera or other device capable of communicating both ways over the Internet or other appropriate communications network. In one example, the imaging device (104) may be integrated within the WSIS (102). The enterprise database (106) is capable of storing the image processing data in the respective datastore or dataset. The enterprise database (106) is configured to store the plurality of acquired source images, transformed images of different phases of WSI creation etc., wherein plurality of source images related to a tile are stored in a data structure indexed by the tile. The WSIS (102) further stores images contained in tiles of the one or more levels of the pyramidal format. In one example, the enterprise database (106) may be integrated within the WSIS (102). The enterprise database (106) may be configured, for example, as a standalone data store or as a cloud data storage as illustrated. In another example, the enterprise database (106) may be integrated within the WSIS (102).

The WSIS (102) may be configured as a cloud-based implementation server or as a standalone server. In one embodiment, the WSIS (102) comprises a processor (112) and a memory (114) coupled to the processor (112) that stores processor-executable instructions. The WSIS (102) further comprises one or more modules configured to perform whole slide imaging of the acquired source images. In one embodiment, the one or more modules include a registration module (116), a transformation module (118), a computation module (120), and a level creation module (122). The WSIS (102) is configured to generate a whole slide image (WSI) by creating plurality of tiles in plurality of levels in a pyramidal format. In one embodiment, the base or base level of the pyramidal format represents the WSI with highest possible scale and/or resolution and subsequent higher levels of the pyramid represent the plurality of WSIs with relatively lower scale and/or resolution. The tiles in each level of the pyramid are designed to map one or more source images, wherein the tile size varies from one level to another. The WSIS (102) unites the creation of WSI and tiled pyramidal format by simultaneously decomposing the entire task into plurality of operations at the level of individual tiles in the pyramid. Such operations are independent and require little memory and processing time, wherein such operations can be either run serially or in parallel. Therefore, the WSIS (102) provides more scalability to create arbitrarily large WSI images.

The WSIS (102) comprises the processor (112) and the memory (114). The imaging device (104) is coupled with the processor (112) and the memory (114) via the network (110). The imaging device (104) is configured to acquire plurality of source images from the glass slide stained with the test sample and stores the digitally acquired images of the test sample in the enterprise database (106).

The WSIS (102) may be a typical whole slide imaging system as illustrated in Figure 2. In one embodiment, the WSIS (102) includes data (204) and modules (206). In one embodiment, the data (204) can be stored within the memory (114). In one example, the data (204) may include image data (208), computational data (212), transformation data (214), level data (216), and other data (210). In one embodiment, the data (204) can be stored in the memory (114) in the form of various data structures. Additionally, the aforementioned data can be organized using data models, such as relational or hierarchical data models. The other data (210) may be also referred to as reference repository for storing recommended implementation approaches as reference data. The other data (210) may also store data, including temporary data, temporary files, predefined image size and overlap offset for different levels generated by the modules (206) for performing the various functions of the WSIS (102).

The modules (206) may include, for example, an image capturing module (218), the registration module (116), the computation module (118), the transformation module (120), the level creation module (122) and a display module (220). The modules (206) may also comprise other modules (222) to perform various miscellaneous functionalities of the WSIS (102). It will be appreciated that such aforementioned modules may be represented as a single module or a combination of different modules. The modules (206) may be implemented in the form of software performed by the processor, hardware and/or firmware.

In one embodiment, the image capturing module (218) enables the imaging device (104) to acquire plurality of images of the sample stained in the glass slide. The image capturing module (218) determines the fundamental quality checks such as blurriness of image, adequate illumination of the acquired images etc., of the acquired images and controls the imaging device (104) to recapture one or more source images upon determining the inappropriate quality. The image capturing module (218) further stores the captured source images in one or more respective datasets as the image data (208).

The registration module (116) retrieves plurality of source images acquired by the imaging device (104) from the image data (208) and registers the acquired source images by performing pairwise registration and global alignment for all the acquired source images to obtain registered images. Pairwise registration is a process of aligning pair of images of a similar scene, wherein pairwise registration process designates one image among the pair of images as a reference image and applies geometric transformation to the other image so that the other image aligns with the reference image. In one embodiment, the registration module (116) retrieves each pair of acquired source images from the image data (208) and computes the relative translation between the pair of source images based on the correlation between pair of source images. Upon computing relative translation, the registration module (116) determines the absolute translation with respect to a fixed coordinate system, wherein the fixed coordinate system can be termed as WSI coordinate system. In an example, the WSI coordinate system is the top left comer of the first image and absolute translation is the translation computed for each source image with respect to the WSI coordinate system. Therefore, the global position of each source image is determined based on the image position with respect to the WSI coordinate system. The registration module (116) computes required scaling and rotation for each of the source images. The registration module (116) determines the size of the WSI by estimating the coordinate of the bottom right corner of the bottom most right image among the acquired source images based on the determined global position of the source images. The registration module (116) further stores the determined transformational information into the transformation data (214).

The computation module (118) computes total number of images in the base level of pyramid based on the determined WSI size and the size of each source image of the respective level. Upon determining the total number of images in the base level of pyramid, the computation module (118) determines dependencies of each tile in the base level of the pyramid based on one or more source images. The computation module (118) can also determine dependencies for each tile in subsequent higher level of the pyramid as well. The computation module (118) computes a plurality of overlapping source images for each of the tiles of the base level of the pyramid in order to determine dependencies of each tile.

In one embodiment, the plurality of overlapping source images with respect to each tile are determined by using one of plurality of pixel matching techniques. In another embodiment, the computation module (118) determines the bounding box in WSI coordinate system for each of the source images, wherein the bounding box is computed based on the predefined fixed image size, overlap offset of the respective level of pyramid, and the transformational information as determined by the registration module (116). Upon determining the bounding boxes of all the registered source images, the computation module (118) further determines the overlapping source images with respect to a tile by matching the bounding box of the tile with the respective bounding boxes of the source images. The computation module (118) stores the identity of the overlapping source images related to each tile in data structures indexed by the respective tile, wherein the data structures are stored as the computational data (212) in the enterprise database (106). As an example, there is a pair of images i.e., image A and image B, where image A is axis parallel and image B is not axis parallel. In WSI coordinate system, the columnar position of image A is (10, 20) i.e., image A spans between column 10 and column 20 of the WSI coordinate system and columnar position of image B is (18, 30) i.e., image B spans between column 18 and column 30. Therefore, the bounding boxes of image A and image B are [10, 20] and [18, 30] respectively. Upon comparing the bounding boxes of image A and image B, the overlap of image B on image A can be easily determined based on the overlapping columnar position of image A and image B.

The computation module (118) further generates one or more tiles in the base level of the pyramid based on a predefined tile size for the base level of the pyramid, determined dependencies of each tile, and size of each of the source images. The tiles are the individual source images that are processed along with the adjacent source images to form the part of the WSI.

The transformation module (120) retrieves the plurality of overlapping source images with respect to one or more tiles and applies the transformation as determined by the registration module (116) to the overlapping source images of each tile. In one embodiment, the transformation can be an affine transformation. An affine transformation is a linear mapping method that preserves points, straight lines, and planes. As an example, sets of parallel lines remain parallel after affine transformation. The affine transformation technique is typically used to correct geometric distortions or deformations that occur with imperfect imaging angle. In another embodiment, the transformation is decomposed in three different parts such as scaling, rotation, and translation, wherein scaling and rotation are specific to respective image. Image scaling is the process of resizing a digital image, where scaling down makes an image smaller and scaling up enlarges an image.

Image rotation is a process of matching and aligning an image with respect to a reference plane, wherein the rotation is performed based on the input image, rotation angle, and a rotation point. Image translation is the process of moving an image to a certain distance either vertically or horizontally, wherein the image translation does not include any rotation or re-sizing of image. The scaling and rotation are applied only once to each of the source images so that the scaled and rotated source images can be reused in rest of the image processing, thereby reducing the computational complexity of the whole slide image processing method. Furthermore, one or more translations are applied on the source images in different phases of image processing, wherein the scaling and rotation are applied only once to each source image thereby further reducing the computational complexity of the whole slide image processing method.

The transformation module (120) retrieves the one or more overlapping source images for each tile from the computational data (212) in the enterprise database (106) and determines whether each of the overlapping source images is scaled and rotated. The transformation module (120) applies scaling and rotation transformation to overlapping source images that are not scaled and rotated and applies translation transformation to each of the overlapping source images in order to align the overlapping source images with respect to each tile.

In an embodiment, the transformation module (120) further blends all the transformed images, compensating the shading difference between plurality of tiles, wherein each tile is configured to map one or more transformed images. Shading difference happens due to variances in pixel intensity of plurality of acquired images and results into clearly visible brightness difference in the overlapping areas of the adjacent tiles. The image blending process combines the colours of corresponding pixels of the overlapping images. The transformation module (120) stores the transformed and blended source images with respect to each tile of the base level of pyramid in data structures indexed by the respective tile, wherein the data structures are stored as the level data (216). The transformed and blended source images enable the creation of WSI by eliminating shading difference between plurality of tiles.

Upon processing the base level of pyramidal image tiles, the level creation module (122) recursively generates plurality of higher levels in the pyramid based on the preceding level of the pyramid by manipulating the size of tile and the source images. The level creation module (122) doubles the size of the tile and reduces the size of the source images by half in each higher level with respect to the preceding lower level. The base level of the pyramid contains the images with highest resolution and the subsequent higher level of pyramid contains the images with relatively lesser resolution. The level creation module (122) generates the plurality of pyramid levels until a WSI size of lxl is generated, wherein the WSI size of lxl is the smallest possible resolution of a WSI. The plurality of levels of pyramidal images enables the viewer of the WSI to enlarge or reduce the viewing area of the whole slide image in one or more scales as required. The level creation module (122) stores each level of the pyramid as a separate independently accessible component in the level data (216) so that the tiles of the WSI can be transmitted without having any memory and computation overhead. In the process of pyramid generation, blending and transformation of one image is independent of other images at the same level of the pyramid. Therefore, such blending and transformation process can be distributed over the network for parallel processing of different images of the same level of pyramid based on the network speed and computation cost.

Upon creation of all the levels of the pyramid, a user can view different areas of whole slide image of the sample with different zoom as required. The display module (220) receives information related to specific areas of the whole slide image to display and required zoom factor from the user. The display module (220) retrieves one or more tiles associated with whole slide image and the one or more source images for each tile from the level data (216) of the enterprise database (106) based on the received area information and zoom factor and displays the specific areas of the whole slide image in desired zoom factor to the user.

Figure 3A-3B illustrates an exemplary representation of pyramidal format of a whole slide image stored in different scales in accordance with an embodiment of the present disclosure.

As illustrated in Figure 3A, plurality of WSI scales are shown in a pyramidal format (310). The pyramidal format (310) comprises five different levels and a plurality of tiles (312) in each level. The base level (Level 0) of the pyramidal format comprises the transformed acquired images with highest resolution and enables the generation of WSI with highest resolution. The subsequent higher levels (Level 1, Level 2, Level 3, Level 4) comprises the images with lesser resolution with respect to the preceding lower level. The highest level of the pyramid i.e., Level 4 comprises an image of WSI with size of lxl. Therefore, the plurality of levels with different scales enables the viewer of the WSI to deep Zoom i.e., zoom in or zoom out the specific area of a whole slide image as required. Figure 3B represents the WSI of “ Mona Lisa ” in four different levels of a pyramid (320), where the base level comprises the highest number of tiles and respective images with highest resolution. The subsequent higher levels of the pyramid comprise the images that are part of “ Mona Lisa ” WSI with lesser resolution with respect to the preceding lower level. Figure 3B further describes the doubling up the tile size and descaling the image size by half in each succeeding higher level. The highest level of the pyramid comprises the images that can combinedly generate a WSI of size lxl.

Figure 4 illustrates a flowchart showing a method for creating a pyramid of whole slide images in accordance with some embodiments of the present disclosure.

As illustrated in Figure 4, the method (400) comprises one or more blocks implemented by the processor (116) to generate whole slide image by creating plurality of tiles in plurality of levels in a pyramidal format using WSIS (102). The method (400) may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, and functions, which perform specific functions or implement specific abstract data types.

The order in which the method (400) is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof.

At block (402), a set of source images of a sample from a glass slide stained with the sample are received. The imaging device (104) acquires plurality of source images of predefined size from a glass slide stained with the sample. In one embodiment, the user of WSIS (102) can feed the plurality of source images to the WSIS (102), wherein the plurality of source images is acquired separately using external imaging device. The WSIS (102) receives the plurality of acquired source images and stores the received source images in one or more respective datasets as the image data (208) of WSIS (102).

At block (404), the required transformation of each of the acquired source images are computed. In one embodiment, the registration module (116) computes relative translation for each adjacent pair of source images by performing pairwise registration, wherein the pairwise registration is performed using conventional transformation technique such as Fourier transform-based correlation method. Upon determining the relative translation, the registration module (118) computes the global position of each source image based on the computed pairwise translations of the plurality of acquired source images by using a global optimizer, wherein the global position is the position of an image with respect to a fixed coordinate system i.e., WSI coordinate system. As an example, global position of an image is determined with respect to the topmost left corner of the first image of the WSI. Therefore, the registration module (116) registers the acquired source images by performing pairwise registration and global alignment. The registration module (116) also computes required scaling and rotation for each of the source images. The registration module (116) further stores the determined transformational information into the transformation data (214). The transformation module (120) applies the determined translation to the acquired source images so that the respective images can be part of WSI. In another embodiment, the transformation module (120) splits the transformation process into three parts such as pure scaling, rotation, and translation. The scaling and rotation parts are applicable for specific image only, wherein the translation part is executed with reference to one or more adjacent source images. Therefore, the scaling and rotation can be applied only once instead of applying all the three transformation parts to the plurality of acquired images every time the acquired images are required to be transformed, wherein the one time application of scaling and rotation transformation reduces the computational complexity of the process in a great extent.

The computation module (118) determines the size of WSI by estimating the coordinate of the bottom right corner of the most bottom right image of the registered source images according to the global position as determined by the registration module (116).

At block (406), dependencies for each tile of a level of the pyramid is determined based on one or more source images. The computation module (118) computes the number of images in the base level of the pyramid based on the image size and determined WSI size. Upon determining the total number of images in the base level of pyramid, the computation module (118) determines dependencies of each tile in the base level of the pyramid based on one or more source images. The computation module (118) can also determine dependencies for each tile in subsequent higher level of the pyramid as well. The computation module (118) further computes plurality of overlapping source images with respect to each tile of the base level of the pyramid in order to determine dependencies of each tile. In one embodiment, the plurality of overlapping source images with respect to a tile of the WSI are determined by using pixel matching technique. In another embodiment, the computation module (118) determines the bounding box in WSI coordinate system for each of the source images, wherein the bounding box is computed based on the predefined fixed image size and overlap offset of the respective level of the pyramid. Upon determining the bounding boxes of all the images, the computation module (118) further determines the overlapping source images with respect to a tile by matching the bounding box of the tile with the respective bounding boxes of the source images. The computation module (118) stores the overlapping source images related to each tile in data structures indexed by the respective tiles, wherein the data structures are stored as the computational data (212).

In an example, W_j is a tile in the pyramid. The source image at column i and row j of the WSI coordinate, the coordinates of top left and bottom right comer are (is + (i - l)o; js + (j - l)o) and ((i + l)s + (i + l)o; (j + l)s + (j + l)o) respectively, wherein s is image size and o is overlap offset. These coordinates are symmetric in row and column. The column coordinates of the top left and bottom right comer are only considered for clearly representing as an example. Thus, the bounding box wij can be represented by its bounding box [wij] as represented in equation 1,

[wij] = (is + (i - l)o; (i + l)s + (i + l)o) . (1)

To compute overlap Aij the [t_x{a_xy\ that has pixels that fall inside Wij, is required to find, wherein t_x is applied transformation and a_x is a set of source images. Let us use l_x and r_x to represent the left most and right most columns of [ _*(<_¾)]. Then for a_x to overlap Wij, equation (2) and (3) are necessary to be satisfied. lx £ (i + l)s + (i + l)o . (2)

The equations (2) and (3) are further combinedly represented as equation (4) below: lx T_y + O

- l £ i £

S + O S + 0 . (4)

Consequently, the equation (5) can be derived as below: lx r_Y + o a_x e A _j V i, i if - 1 < i < . (5)

^{J J 1} s + o s + o

Finally, the overlapping images can be represented as in equation (6) illustrated below:

At block (408), one or more tiles in the level of the pyramid are generated based on tile size and determined dependencies. The computation module (118) further generates one or more tiles in the base level of the pyramid based on a predefined tile size for the base level of the pyramid, determined dependencies of each tile, and size of each of the source images. The tiles are the individual source images that are processed along with the adjacent source images to form the part of the WSI.

At block (410), transformation is applied to each of the one or more source images identified based determined dependencies for each tile for generating a WSI of the level. The transformation module (120) retrieves plurality of overlapping source images with respect to one or more tiles and applies the transformation as determined by the computation module (116) to the overlapping source images for each tile. In one embodiment, the transformation module (120) applies the affine transformation to the overlapping source images. In another embodiment, the transformation module (120) applies all the parts of the transformation such as scaling, rotation, and translation to the overlapping source images. The transformation module (120) can retrieve a source image multiple times as a source image can overlap plurality of tiles and applies all the parts of transformation upon the first retrieval of the source image. The transformation module (120) applies only the required translation to the source image in the subsequent retrieval of the source image as scaling and rotation transformations are specific to individual image and can be applied only once. The reduced application of all the transformation parts eliminates the requirement of large RAM, large storage and complex computational capability of the processor.

The transformation module (120) further blends all the transformed images, wherein blending process compensates the shading difference between plurality of tiles. Shading difference happens due to variances in pixel intensity of plurality of acquired source images and results into clearly visible brightness difference in the overlapping areas of the adjacent tiles. The transformation module (120) further stores the transformed and blended source images with respect to each tile of base level of pyramid in data structures indexed by the respective tile, wherein the data structures are stored as the level data (216). The transformed and blended source images enable the creation of WSI by eliminating shading difference between plurality of tiles.

At block (412), WSI for higher level of the pyramid is generated based on a preceding base level. The level creation module (122) generates plurality of higher levels of the pyramidal format based on the preceding base level of the pyramid. The level creation module (122) generates the higher levels of the pyramid based on the preceding lower level in a recursive manner until the level creation module generates a WSI of size lxl. In one embodiment, the level creation module (122) generates each higher level by doubling the tile size and descaling size of the source images by half with respect to the preceding lower level. Upon adjusting the tile size and image size, the level creation module (122) follows the steps of (406, 408, and 410) to generate the plurality of images that combinedly represents the WSI of the respective level. The level creation module (122) further stores the processed images in separate independently accessible datasets as the level data (216), so that the WSI images can be transmitted or viewed with low memory requirement. Therefore, the pyramid levels having images of different scales enable the viewer of the WSI to enlarge or descale the WSI as required.

Upon creation of all the levels of the pyramid, a user can view different areas of whole slide image of the sample with different zoom as required. The display module (220) receives information related to specific areas of the whole slide image to display and required zoom factor from the user. The display module (220) retrieves one or more tiles associated with whole slide image and the one or more source images for each tile from the level data (216) of the enterprise database (106) based on the received area information and zoom factor and displays the specific areas of the whole slide image in desired zoom factor to a user.

For example, when the user i.e., authorized person of a pathology wishes to view the whole slide image of a glass slide stained with test sample, the user places the glass slide under the scanner and the WSIS (102) acquires 20 images of the entire glass slide by using the imaging device (104) and stores the acquired images into a dataset of the enterprise database (106). The WSIS (102) registers the 20 images to determine the respective transformation of the images and performs global alignment based on the determined transformation. The WSIS (102) decomposes the transformation into three different parts i.e., scaling, rotation and translation. The WSIS (102) applies the scaling and rotation transformation to the source images only once to reduce the computational complexity. Upon aligning the images, the WSIS (102) determines the size of the WSI as 4000 units in column width based on the global position of the acquired images, wherein each of the acquired images occupies 200 units.

The WSIS (102) further determines the number of images based on the image size and the WSI size. Upon determining the number of images, the WSIS (102) computes the bounding box of each images based on the predefined fixed image size and overlap offset of the base scale of the pyramid. The WSIS (102) further computes the plurality of source images overlapping each tile of the acquired 20 images stores the source images with respect to each tile in a data structure indexed by the tile. The WSIS (102) retrieves the source images with respect to each tile and apply required transformation to the source images to accommodate the tile in a specific tile of the base level of the pyramid. In the process of transformation, the WSIS (102) checks the source images for required scaling and rotation transformation and applies such transformation to the source images in case such transformations are not applied. The WSIS (102) blends the source images with respect to each tile and saves the respective area in the enterprise database (106). The WSIS (102) further generates the higher levels of the pyramid by doubling the tile size and descaling each image size of 20 source images by half with respect to the saved tile size and image size of the preceding lower level of the pyramid, wherein the steps of creating a level in the pyramid is analogous to the steps of creating base level of the pyramid. Therefore, the subsequent higher level of the pyramid computes each image size as 100 units and each tile size as 400 units, wherein the WSI size in the subsequent higher level reduces to 2000 units. The authorized person of the pathology zooms in the WSI while viewing the WSI and upon determining the zoom in request the WSIS (102) retrieves the stored images from the lower level of the pyramid. The authorised person can further requests for zoom out the WSI for viewing an overall view and the WSIS (102) provides the images from the higher level of the pyramid.

In an embodiment, the computer system (500) may be whole slide imaging system (102), which is used for generating whole slide image from a plurality of acquired images of a glass slide. The computer system (500) may include a central processing unit (“CPU” or “processor”) (508). The processor (508) may comprise at least one data processor for executing program components for executing user or system-generated business processes. The processor (508) may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc.

The processor (508) may be disposed in communication with one or more input/output (I/O) devices (502 and 504) via I/O interface (506). The I/O interface (506) may employ communication protocols/methods such as, without limitation, audio, analog, digital, stereo, IEEE-1394, serial bus, Universal Serial Bus (USB), infrared, PS/2, BNC, coaxial, component, composite, Digital Visual Interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array (VGA), IEEE 802.n /b/g/n/x, Bluetooth, cellular (e.g., Code-Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System For Mobile Communications (GSM), Long-Term Evolution (LTE) or the like), etc.

Using the I/O interface (506), the computer system (500) may communicate with one or more I/O devices (502 and 504). In some implementations, the processor (508) may be disposed in communication with a communication network (110) via a network interface (510). The network interface (510) may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10/100/1000 Base T), Transmission Control Protocol/Internet Protocol (TCP/IP), token ring, IEEE 802.1 la b/g/n/x, etc. Using the network interface (510) and the communication network (110), the computer system (500) may be connected to the enterprise database (106), the WSIS (102) and the imaging device (104).

The communication network (110) can be implemented as one of the several types of networks, such as intranet or any such wireless network interfaces. The communication network (110) may either be a dedicated network or a shared network, which represents an association of several types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Intemet Protocol (TCP/IP), Wireless Application Protocol (WAP), etc., to communicate with each other. Further, the communication network (110) may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, etc.

In some embodiments, the processor (508) may be disposed in communication with a memory (530) e.g., RAM (514), and ROM (516), etc. as shown in Figure 5, via a storage interface (512). The storage interface (512) may connect to memory (530) including, without limitation, memory drives, removable disc drives, etc., employing connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE- 1394, Universal Serial Bus (USB), fiber channel, Small Computer Systems Interface (SCSI), etc. The memory drives may further include a drum, magnetic disc drive, magneto-optical drive, optical drive, Redundant Array of Independent Discs (RAID), solid-state memory devices, solid-state drives, etc.

The memory (530) may store a collection of program or database components, including, without limitation, user/application (518), an operating system (528), a web browser (524), a mail client (520), a mail server (522), a user interface (526), and the like. In some embodiments, computer system (500) may store user/application data (518), such as the data, variables, records, etc. as described in this invention. Such databases may be implemented as fault- tolerant, relational, scalable, secure databases such as Oracle or Sybase.

The operating system (528) may facilitate resource management and operation of the computer system (500). Examples of operating systems include, without limitation, Apple Macintosh ™ OS X ™, UNIX ™, Unix-like system distributions (e.g., Berkeley Software Distribution (BSD), FreeBSD ™, Net BSD ™, Open BSD ™, etc.), Linux distributions (e.g., Red Hat™, Ubuntu ™, K-Ubuntu ™, etc.), International Business Machines (IBM ™) OS/2 ™, Microsoft Windows ™ (XP ™, Vista/7/8, etc.), Apple iOS ™, Google Android ™, Blackberry ™ Operating System (OS), or the like. A user interface may facilitate display, execution, interaction, manipulation, or operation of program components through textual or graphical facilities. For example, user interfaces may provide computer interaction interface elements on a display system operatively connected to the computer system (500), such as cursors, icons, check boxes, menus, windows, widgets, etc. Graphical User Interfaces (GUIs) may be employed, including, without limitation, Apple ™ Macintosh™ operating systems’ Aqua™, IBM ™ OS/2 ™, Microsoft™ Windows ™ (e.g., Aero, Metro, etc.), Unix X-Windows ™, web interface libraries (e.g., ActiveX, Java, JavaScript, AJAX, HTML, Adobe Flash, etc.), or the like.

The illustrated steps are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments. Also, the words "comprising," "having," "containing," and "including," and other similar forms are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.

Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., are non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the embodiments of the disclosure is intended to be illustrative, but not limiting, of the scope of the disclosure.

With respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity.

Claims

The Claim:

1. A method of creating a pyramid of whole slide images of a sample, method comprising: receiving a set of source images of the sample acquired from a glass slide stained with the sample; computing transformation required for each of the source images to accommodate in a Whole Slide Image (WSI); for each level in the pyramid that comprises a base level and at least one higher level, generating a WSI of each level by: determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile; generating one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images; and applying transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

2. The method as claimed in claim 1, further comprising: blending the transformed one or more source images to compensate shading difference between the tiles; storing the transformed one or more source images indexed by each tile of the one or more tiles of respective level of the pyramid in a datastore; and retrieving the stored one or more source images for each tile while displaying a particular area of the whole slide image in a particular resolution to a user.

3. The method as claimed in claim 1, wherein computing transformation required for each of the source images comprises steps of: computing relative translation for each adjacent pair of source images; computing required scaling and rotation for each of the source images; determining global position of each of the source images with respect to a fixed coordinate system; and estimating the size of WSI with respect to the fixed coordinate system.

4. The method as claimed in claim 3, wherein the size of WSI is estimated based on the coordinate of bottom right corner of most bottom right source image among the set of source images in the fixed coordinate system.

5. The method as claimed in claim 1, wherein determining dependencies for each of the one or more tiles comprises steps of: generating a bounding box for each of the source images based on size of source image and a predefined overlap offset, wherein the overlap offset indicates an overlap between a source image and a tile; determining one or more overlapping source images for each of the one or more tiles in the WSI; and storing the one or more overlapping source images indexed by each of the one or more tiles in the datastore.

6. The method as claimed in claim 1, wherein applying transformation to each of the one or more source images comprises the steps of: retrieving the one or more overlapping source images for each tile; determine whether each of the overlapping source images is scaled and rotated; applying scaling and rotation transformation to overlapping source images that are not scaled and rotated; and applying translation transformation to each of the overlapping source images in order to align the overlapping source images with respect to each tile.

7. The method as claimed in claim 1, wherein WSI of the higher levels of the pyramid are recursively generated based on the WSI in the base level of the pyramid until the size of WSI is lxl, wherein WSI in each of the higher levels of the pyramid is generated by setting size of each tile as double and downscaling size of source image by half.

8. A system for creating a pyramid of whole slide images of a sample, the system comprising: an imaging device (104); a processor (112); a memory (114) communicatively coupled with the processor (112), wherein the processor (112) is configured to: receive a set of source images of the sample acquired by the imaging device (104) from a glass slide stained with the sample; compute transformation required for each of the source images in order to accommodate the source images in a Whole Slide Image (WSI); for each level in the pyramid that comprises a base level and at least one higher level, generate a WSI by: determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile; generating one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images; and applying transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

9. The system as claimed in claim 8, wherein the processor (112) is further configured to: blend the transformed one or more source images to compensate shading difference between the tiles; store the transformed one or more source images indexed by each tile of the one or more tiles of respective level of the pyramid in a datastore; and retrieve the stored one or more source images for each tile while displaying a particular area of the whole slide image in a particular resolution to a user.

10. The system as claimed in claim 8, wherein the processor (112) is configured to compute transformation required for each of the source images by performing the steps of: computing relative translation for each adjacent pair of source images; computing required scaling and rotation for each of the source images; determining global position of each of the source images with respect to a fixed coordinate system; and estimating the size of WSI with respect to the fixed coordinate system.

11. The system as claimed in claim 8, wherein the processor (112) estimates the size of WSI based on the coordinate of bottom right comer of most bottom right source image among the set of source images in the fixed coordinate system.

12. The system as claimed in claim 8, wherein the processor (112) is configured to determine dependencies for each of the one or more tiles by performing the steps of: generating a bounding box for each of the source images based on size of source image and a predefined overlap offset, wherein the overlap offset indicates an overlap between a source image and a tile; determining one or more overlapping source images for each of the one or more tiles in the WSI; and storing the one or more overlapping source images indexed by each of the one or more tiles in the datastore.

13. The system as claimed in claim 8, wherein the processor (112) is configured to apply transformation to each of the one or more source images by performing the steps of: retrieving the one or more overlapping source images for each tile; determine whether each of the overlapping source images is scaled and rotated; applying scaling and rotation transformation to overlapping source images that are not scaled and rotated; and applying translation transformation to each of the overlapping source images in order to align the overlapping source images with respect to each tile.

14. The system as claimed in claim 8, wherein the processor (112) recursively generates WSI of higher levels of the pyramid based on the WSI in the base level of the pyramid until the size of WSI is lxl, wherein the WSI in each of the higher levels of the pyramid is generated by setting size of each tile as double and downscaling size of source image by half.

15. A non-transitory computer-readable storage medium that stores instructions executable by a processor that, in response to execution of the instructions, cause the processor to perform operations comprising: receiving a set of source images of the sample acquired from a glass slide stained with the sample; computing transformation required for each of the source images to accommodate in a Whole Slide Image (WSI); for each level in the pyramid that comprises a base level and at least one higher level, generating a WSI of each level by: determining dependencies of each tile in the level based on one or more source images, wherein each level comprises at least one tile; generating one or more tiles based on a predefined tile size for the level of pyramid, determined dependencies of each tile, and size of each of the source images, wherein each of the one or more tiles maps to one or more source images; and applying transformation to each of the one or more source images identified based on determined dependencies for each tile, wherein transformation include scaling, rotation and translation.

16. The non-transitory computer-readable storage medium as claimed in claim 15, further store executable instructions that cause the processor to perform operations comprising: blending the transformed one or more source images to compensate shading difference between the tiles; storing the transformed one or more source images indexed by each tile of the one or more tiles of respective level of the pyramid in a datastore; and retrieving the stored one or more source images for each tile while displaying a particular area of the whole slide image in a particular resolution to a user.

17. The non-transitory computer-readable storage medium as claimed in claim 15, wherein the stored executable instructions further cause the processor to compute transformation required for each of the source images by: computing relative translation for each adjacent pair of source images; computing required scaling and rotation for each of the source images; determining global position of each of the source images with respect to a fixed coordinate system; and estimating the size of WSI with respect to the fixed coordinate system, wherein the size of WSI is estimated based on the coordinate of bottom right corner of most bottom right source image among the set of source images in the fixed coordinate system.

18. The non-transitory computer-readable storage medium as claimed in claim 15, wherein the stored executable instructions further cause the processor to determine dependencies for each of the one or more tiles by: generating a bounding box for each of the source images based on size of source image and a predefined overlap offset, wherein the overlap offset indicates an overlap between a source image and a tile; determining one or more overlapping source images for each of the one or more tiles in the WSI; and storing the one or more overlapping source images indexed by each of the one or more tiles in the datastore.

19. The non-transitory computer-readable storage medium as claimed in claim 15, wherein the stored executable instructions further cause the processor to apply transformation to each of the one or more source by: retrieving the one or more overlapping source images for each tile; determine whether each of the overlapping source images is scaled and rotated; applying scaling and rotation transformation to overlapping source images that are not scaled and rotated; and applying translation transformation to each of the overlapping source images in order to align the overlapping source images with respect to each tile.

20. The non-transitory computer-readable storage medium as claimed in claim 15, wherein the stored executable instructions further cause the processor to recursively generate WSI of the higher levels of the pyramid based on the WSI in the base level of the pyramid until the size of WSI is lxl, wherein WSI in each of the higher levels of the pyramid is generated by setting size of each tile as double and downscaling size of source image by half.