JP2023123838A - Continuous image processing - Google Patents

Abstract

To provide a method for querying, analyzing and processing data.SOLUTION: A method for searching a database of images based on a query comprises: if a magnification level of the query is greater than a first threshold, returning a first list of result tiles satisfying the query at the magnification level of the query; if the magnification level is less than or equal to the first threshold, retrieving tiles at a next lower magnification level and returning a second list of result tiles satisfying the query at the next lower magnification level; and processing each list of result tiles, the processing including, for each result tile: adding the result tile to a subset of result tiles; if the number of result tiles in the subset is greater than or equal to a second threshold, recursively searching the subset; saving results of each recursive search of the subset in a remaining subset; recursively searching the remaining subset; and saving results of the search of the remaining subset.SELECTED DRAWING: Figure 1

Description

Related application

This application claims priority from U.S. Provisional Patent Application No. 61/905,027 entitled "Continuous Image Analytics," filed November 15, 2013, and "Continuous Image Analytics," filed November 15, 2014. and claims priority from PCT International Application No. PCT/US14/65850 entitled "Analytics". The entire teachings of each application are incorporated herein by reference.

Copyright Reservation

A portion of the disclosure of this document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by any person of this application or the disclosure of this application as it appears in the patent file or records of the Patent Office, but otherwise reserves all copyright rights whatsoever. do.

Methods, systems, and sets of computer-readable instructions in storage media (eg, non-transitory storage media) are provided for querying, analyzing, and processing data. Specifically, methods, systems and storage media (e.g., non-transient provides a set of computer readable instructions on a storage medium).

Relevant data can be retrieved from the data repository by requesting it with a query. However, in existing database-query systems, there is a semantic gap between the user's conceptual expectations (eg, the conceptual expectations conveyed in the query) and the low-level representation of the data. Specifically, in the case of tissue imaging, there is a semantic semantic assignment of meaning between the low-level representation of a digital image of microscopic tissue and the user's intent (i.e., the intent conveyed in the query). A gap exists. The magnitude of this semantic gap is so large that it makes general application of content-based image retrieval techniques impossible. Specifically, a query that is too broad can return irrelevant results, and a query that is too specific can eliminate relevant results.

The semantic gap problem is due to the complexity, variability and size of the data. These factors complicate the algorithm's differentiating means (e.g., work against the differentiating means), and in the extreme case, the error model may change the pattern of the image data as the algorithm is extended beyond the constraint coverage. It could end up dominating the model. Another problem is the practical approximate assumptions used when applying algorithms to large amounts of image data. An approximation here refers to a subset or summary of the image data to make the algorithm computationally tractable. In the case of tissue image data, the scale of the data is extremely wide, which not only makes distinguishing features difficult to understand, but also makes different meanings of distinguishing features at different scales.

The present invention is a computer-implemented method of query-based searching of a database of images, comprising: responsive to determining that a magnification level of said query is greater than a first threshold; returning a first list of result tiles that satisfy the query at a magnification level; and retrieving a tile with the next lowest magnification level in response to determining that the query's magnification level is less than or equal to the first threshold. and returning a second list of result tiles that satisfy the query at the next lower magnification level; and processing the result tiles of each list, wherein, for each result tile, the result tile adding to a subset of tiles; recursively searching the subset in response to determining that the total number of result tiles in the subset is greater than or equal to a second threshold; saving the results of a search to a remaining subset, recursively searching the remaining subset, and saving the results of the search for the remaining subset; provide a way.

In one embodiment, each magnification level corresponds to a level in the quadtree, and each level in said quadtree contains tiles representing image results. In one embodiment, a child tile has an association with a parent tile. In one embodiment, said parent tile is at least one of down-sampling and low-pass spatial filtering of said at least one corresponding child tile. In one embodiment, the step of retrieving the next lower magnification level tile includes generating children at the next lower level in the quadtree. In one embodiment, the query includes at least one of a minimum number of results threshold and a maximum number of results threshold. In one embodiment, the threshold number of results is based on system resources. In one embodiment, the query includes an image and the magnification level of the query is the magnification level of that image. In one embodiment, a result tile is determined based on at least one of a scale factor of the result tile, an image of the query, a filename of an index associated with the result tile, a result size, and an index type. Included in the returned list. In one embodiment, the query includes a time limit within which to perform the search. In one embodiment, the query includes a quality level threshold. In one embodiment, the first predetermined threshold is defined such that the method terminates in response to determining that the number of search results falls below a certain number. In one embodiment, said first predetermined threshold is defined to correspond to a number of magnification levels. In one embodiment, said first predetermined threshold is defined such that no depth-based search is used. In one embodiment, one magnification level has a higher resolution than a lower magnification level. In one embodiment, one magnification level has twice the resolution of each dimension of the next lower magnification level. In one embodiment, the query is updated after returning the first list of result tiles. In one embodiment, the method further includes excluding results from at least one of the first list of result tiles and the second list of result tiles before the step of processing each list of result tiles. including the process of In one embodiment, the process of processing the result tiles of each list further comprises clearing the subset after saving the results of each recursive search, and clearing the results of the recursive search for the remaining subsets. clearing the remaining subset after saving the . In one embodiment, the recursive search is a depth-first search. In one embodiment, the saved results are available before the search is terminated.

In one embodiment, a method and system for performing a recursive search of a tileset based on a query tile includes, for each tile of said tileset, from the next level until the result set is populated. performing the steps of retrieving a set of tiles and adding the next level tileset to the result set; and determining that the magnification level is below the target level, for each tile in the result set, the number of tiles in the subset is a third adding the tile to the subset in response to determining that it is greater than or equal to a threshold; and recursing through the subset in response to determining that the number of tiles in the subset is less than the third threshold. adding results of the recursive search to a provisional result set; clearing the subset; recursively searching the subset; adding results to the provisional result set, clearing the subset, and returning the provisional result set.

In one embodiment, the query tile is included as a first child tile of a first tile in the tileset. In one embodiment, evaluating match quality comprises determining whether a match value of a first tile in the result set compared to the query tile is less than a predetermined value. have. In one embodiment, the predetermined value is 50%. In one embodiment, match quality is based on differences between vectors. In one embodiment, the difference between vectors is based on the distance between the vectors of the respective tiles. In one embodiment, the difference between vectors is based on the mean squared error between the vectors of each tile. In one embodiment, each pixel in a tile has at least one numerical value representing at least one of the color and brightness of that each pixel, and each tile includes all pixels in that tile. a vector of said at least one numerical value for . In one embodiment, the query includes the predetermined value for evaluating match quality. In one embodiment, the returned interim result set is sorted based on matching with the query. In one embodiment, the interim result set is sorted based on how well it matches the corresponding parent tile.

In one embodiment, the computer-implemented processing execution plan includes at least one selectable probe feature specification including at least the spatial location and extent of image features and at least one target specification including a set of images. wherein said set of images includes at least one image of a microscope slide, for generating at least one target designation and correlation samples of said at least one selectable probe feature and said at least one target designation. a traversal plan comprising a comparison order and a comparison operator for the correlation samples, the similarity method, the similarity measure, the at least one selectable probe feature designation, the at least one target designation, and the including said spread of image features. In one embodiment, said traversal plan comprises at least a method of ordering samples and applying a similarity measure to establish a correlation with said at least one selectable probe feature specification, wherein data comprising said correlation is held in persistent computer memory available to the traversal plan such that the processing execution plan is adapted to evaluate correlations by the traversal plan. In one embodiment, the traversal plan divides the samples into statistically uniform sampling with a uniform grid, quadtree decomposition of the slide, embedded zerotree of the slide, exhaustive sampling, sparse ) sampling, and scale-proximity-biased sampling.

In one embodiment, a transitional bias is adaptively applied such that correlation with each data can be predicted using at least one correlation with previously correlated data. In one embodiment, online machine learning is used to bias the sampling and the traversal plan. In one embodiment, relevance feedback from the user is used to bias sampling during execution of the traversal plan. In one embodiment, the traversal plan defines result parameters that can be used to determine the samples returned as part of a result set, the parameters including magnification scale and spatial extent, and the processing execution plan defines the order in which the samples are evaluated, and that order determines the rate at which the samples of the result set are returned.

In one embodiment, trees of scale-based dependencies are defined based on isolation of each evaluation state, and the trees of scale-based dependencies are distributed to separate processing elements for parallel evaluation. In one embodiment, defined partitions of data are provided independently of other data partitions, and intermediate sets of data are generated for partial results. In one embodiment, the divided data and processing specifications are stored on the same storage device. In one embodiment, the split is based on the number of result samples returned per sample evaluation, and the split size is at least one of increasing and decreasing. In one embodiment, a transformation process is applied to at least one image processing transformation on said result set, and the output of said transformation process comprises transformed samples stored in persistent storage.

In one embodiment, the probe and target samples are derived from the available transformed and secondary samples to form a traversal plan in which the samples returned by executing are secondary result samples. selected. In one embodiment, the secondary result set samples are used to adaptively bias a primary traversal plan, and a strong correlation based on the secondary result set determines the primary traversal plan. Indicates that the corresponding sample in the plan should be evaluated preferentially. In one embodiment, the secondary result samples are adaptively biased to further transform the secondary result samples to produce transformed tertiary samples, and the primary result set The adaptive biasing of the secondary result set to also extends to the biasing of the secondary result set by the tertiary result set. In one embodiment, adaptive biasing of upstream and downstream plans is used in a chain. In one embodiment, the adaptive bias application utilizes the topography of the graph.

In one embodiment, a computer-implemented method for continuously processing a repository of image data comprises receiving a query specification containing a request for data; , comparing said query specification with said system specification to determine a domain specification; initiating a query to said repository based on said domain specification; displaying an interactive iterative search of the resulting image data on a graphical user interface; receiving input of the resulting image data via the graphical user interface; and receiving the query. and displaying the updated graphical user interface based on the updated resulting image data. In one embodiment, said repository of image data includes digital microscopy data. In one embodiment, the repository of image data contains tissue image data at a scale such that approximations of the data on the coarse scale side are uncorrelated with the data on the fine scale side. In one embodiment, the successive processing of the image data produces results incrementally such that the results are available before the processing is completely finished. In one embodiment, the query specification implicitly defines data indices and transformations.

In one embodiment, a computer-implemented method of query-based transformation of image data in a data repository comprises receiving a query for data in the data repository, the query comprising at least one a process comprising probe tiles and at least one group of slides on which the query is performed; recursively searching through magnification levels; refining (refining) the query based on the results of the recursive search; and developing a traversal plan for a target slide based on the recursive search. and transforming data based on the data returned by the query, the transform adjusting at least one of individual pel positions of the data and color depth of the data. have to do.

In one embodiment, the overlap is spatially related in response to determining that there is an overlap of at least 256 pixels. In one embodiment, the query includes a search predicate and a query target. In one embodiment, the probe feature specification includes a descriptive search predicate specified as a region of interest, said region of interest comprising locations on the image having a specified extent. In one embodiment, the probe feature specification includes at least one tile having a set of images that the probe feature specification targets in generating search matches. In one embodiment, the traversal plan includes an order in which targets are compared to at least one probe feature specification. In one embodiment, each of said probe feature designation and said target is at least one of a tile, a single image, and a sub-image of a microscope slide image at a magnification level.

FIG. 4 is a flow diagram illustrating a method of interactive data exploration in an exemplary embodiment; FIG. 4 is a flow diagram illustrating a discovery lifecycle in an exemplary embodiment; FIG. 4 is a block diagram illustrating a query unit in an exemplary embodiment; FIG. 4 is a block diagram illustrating an analysis unit in an exemplary embodiment; FIG. 4 illustrates an intermediate stage architecture of a query unit in an exemplary embodiment; FIG. 4 is a block diagram that illustrates components of a repository in an exemplary embodiment; FIG. 4 illustrates an exemplary embodiment; FIG. 4 illustrates an exemplary embodiment; FIG. 4 illustrates an exemplary embodiment; FIG. 4 illustrates an exemplary embodiment; FIG. 4 is a diagram showing an example of a slide layout in one embodiment of the present invention; FIG. 12 illustrates an example of feature comparison for slides or tiles in an exemplary embodiment; FIG. 4 illustrates an example spatial decomposition of tiles in an exemplary embodiment; FIG. 4 illustrates an example scale decomposition of a tile in an exemplary embodiment;

A method, system and computer readable (stored on a storage medium) instructions for continuously processing data (e.g., image data) are provided to address problems arising from semantic gaps. In one embodiment, the method is driven by a query (which represents the user's intent), system functionality, and user guidance by the system. The method provides for interactive and iterative exploration of data through at least one of querying, analyzing and processing data. For example, the present invention provides an image data search focused on the unique requirements of tissue image data and other biological image data. Various uses are conceivable as uses of the present invention. For example, the invention can be applied to any image in any industry, including photography, satellite imagery, and the like.

In one embodiment, the search method and system provide users with immediate feedback on query scope and query results. This makes it easier to refine queries on the fly. The method further allows specifying analysis or processing to be performed on the image data results returned from the query.

In one embodiment, the query specification implicitly defines derived indices and data transformations for the data. In one embodiment, the definition produces results for a query. These results are in response to the results from that definition. In one embodiment, the results are pre-computed and/or computed on demand and/or computed during previous searches. In one embodiment, the results are returned incrementally based on at least one of user experience requirements, user query specification/refinement, and system capabilities. In one embodiment, multiple query, analysis and processing steps are chained together by iterative processing of each of these steps. The combination of these processes forms the processing pipeline.

In one embodiment, system and method components provide means for the system to continuously solve the processing pipeline. In one embodiment, the results of said processing are generated incrementally and provided to the user and/or later stages of said pipeline. This configuration is advantageous because it is not necessary to completely process all data in one pipeline step. For example, the results are provided each time they are found by at least one processor. when a given result is selected as more relevant, the query is updated with this information, and subsequent searches and results by the at least one processor include the updated query. It is said that The at least one processor then continues the search that was initiated. In an alternative embodiment, the search is started anew when the query is changed.

In one embodiment, the system and method includes archiving, storing, forwarding, and analyzing operations and prioritized in that order. Archival operations include retention and replication of data (eg, tissue image data). This ensures that the data can be stored for a long time without being moved frequently. Performing operations on data as it is archived moves computations on the data local to the data, rather than moving the data to the side local to the computations on the data. involves moving. A store operation provides access to data in a multi-scale, decimated representation. Store operations organize data to reduce access latency and manage storage and loading of derived data. Forwarding operations limit the need to transmit data or derived data. For example, a transfer operation delays the transfer of data to downstream processing and performs analysis and transformation of the data local to the data and returns results when the derived data is small.

FIG. 1 is a flow diagram illustrating a method and system 100 for interactive data exploration in an exemplary embodiment. The method and system 100 includes an order of operations for searching the repository for information. At step 102, user specifications are defined based on the user's data extraction requirements. At step 104, the user specifications defined at step 102 are interspersed with domain-specific information in the repository. This generates an associated domain designation in step 106 . A query is initiated against the repository using a combination of user specification (step 102) and domain specification (step 106). This produces query results in step 108 . The results generated are fed into the analysis process. At step 110, the analysis process produces an analysis of the results.

FIG. 6 is a block diagram illustrating components 600 of a repository in an exemplary embodiment. In this example, the repository components include underlying data 602 for the repository. The underlying data 602 is a quadtree decomposition of the slide image data, specifically, each parent layer is sub-sampled once and divided into tiles, each layer corresponding to a component tile of the tree (e.g., decimation). A tile index generation 604 is defined based on at least one feature extraction 608 . Feature extraction 608 is separated into states for each tile. A correlation index 610 is defined based on index similarity to indicate the correlation between two or more tiles based on at least one index 604 . The underlying data 602 is a transformation 606 of the imported data.

FIG. 2 is a flow diagram illustrating a repository discovery lifecycle 200 in an exemplary embodiment. In one embodiment, the search is a combination of query and analysis performed by at least one query unit 202 and at least one analysis unit 216 in sequence. For example, at least one query unit 202 includes a query target. For this query target, an index 208 and/or correlations 210 are identified, eg, based on feature extraction or the like. Index and correlation information is stored in repository 212 . Results 206 obtained via at least one query unit 202 are sent to analysis unit 216 and/or refinement unit 204 . After analysis unit 216 , the results are sent to transformation unit 214 . After conversion unit 214 , the results are sent to repository 212 . Various information stored in repository 212 is sent to at least one query unit 202 . Refinements 204 of results 206 are also sent to the query unit to update the search.

In one embodiment, at least one query unit 202 includes a search predicate and a query target. As used herein, "probe (search)" or "probe feature specification (search feature specification)" refers to the search predicate specified as the region of interest. A region of interest is, for example, a point on an image with a specified extent. As used herein, "targeting" refers to a set of target images (e.g., digital images) that make up the tiles on at least one microscope slide that the probe targets in generating search matches. refers to the set of In one embodiment, once at least one target image or target tile is selected, the selection becomes the probe. As used herein, a "traversal plan" refers to the construction of an order in which targets from a target specification are compared with at least one probe. In one embodiment, each individual comparison is of one probe to one target. In one embodiment, said comparison is of one or more probes to one or more targets. In one embodiment, said probe and said target are at least one of a tile, a single image, and a sub-image of a microscope slide image at a particular scale or magnification. In one embodiment, the traversal plan is a breadth-first traversal for multi-scale quadtree decomposition of microscope slide images. In one embodiment, the traversal plan is a depth-first traversal for multi-scale quadtree decomposition of microscope slide images. In one embodiment, the traversal plan is a depth traversal followed by a breadth traversal. In one embodiment, the traversal plan is a width traversal followed by a depth traversal.

In one embodiment, a user (eg, pathologist, administrator, processor, etc.) selects at least one probe tile that is used to query a repository of tissue images. The user also selects a group of slides for which the query is run. Executing the query extracts features from at least one probe tile image based on the query specification. For example, color histogram feature extraction is utilized. In one embodiment, once a feature is extracted, it is persistently maintained in long-term storage to prevent recomputation of the feature. A collection of at least one feature vector persistently maintained on disk is defined as an index. A feature vector is extracted from the lower magnification scale of the slide that contains the probe tile. In one embodiment, this process recursively lowers the magnification scale until reaching the last level (e.g., the highest level) where overlap with the probe tile is spatially relevant. go back. In one embodiment, this spatial relationship is 256 pixels of overlap. Extracting features from the highest level of probe overlap tiles for an individual probe tile, and extracting features from intermediate magnification level probe overlap tiles for that individual probe tile is defined as the scale probe tileset of . In one embodiment, the collection of all the tilesets for all the probe tiles (collectively, the total probe tileset) is used to generate a traversal plan for the target slide. In one embodiment, the components of this total probe tileset are combined based on their magnification levels. A collection of such constituents (eg, features extracted from a tileset) is used to order the set of extracted features of candidate target tiles in generating a traversal plan. In one embodiment, features from the target tile are extracted at corresponding magnification levels and compared to the probeset. The multiple target candidates are then ordered based on the feature vector similarity measure. In one embodiment, the comparison operator is the L2 norm of the two vectors. In one embodiment, the traversal plan is this ordered list, traversed in order of similarity, with the tile at the next highest magnification defined as the corresponding probeset tile in feature space (e.g., in this example color histogram). After the results are also ordered, recursive processing continues to stronger magnification levels. In one embodiment, the traversal plan is specified as either breadth-first or depth-first so that higher magnification levels are recursively processed before all evaluations of the current magnification level are completed. or whether higher magnification levels are processed recursively after all evaluations of the current magnification level are completed. For example, the depth-first one has the advantage of providing results to the user with low latency, since fewer evaluations are performed.

In one embodiment, results returned from executing the query using the traversal plan are displayed in a user interface. In one embodiment, the user can add or remove probe tiles, add or remove target slides, and enter relevance feedback on the returned results at any time while results are being displayed in the user interface. or you can choose to change the query parameters. In one embodiment, as probe tiles and slides are added/deleted, simple processing is applied to the existing set of probe tile features, thereby making changes to the traversal plan. In one embodiment, the relevance feedback not only changes the traversal order, but also acts as a biasing factor for the similarity measure. The relevance feedback is designated in the user interface as plus (+), which corresponds to positive feedback, or minus (-), which corresponds to negative feedback.

In one embodiment, a user may specify at least one additional query that targets results of a first query during execution of said query using said traversal plan. Specifically, the later query operates on the results of the earlier query and retrieves its result set. In one embodiment, the primary query is based on color histogram feature extraction and the secondary query is a more complex feature-based query (eg, based on texture characterization). In one embodiment, the secondary query is based on texture orientation (eg, histogram feature extraction with sparse Gabor).

In one embodiment, a user specifies analytical transformations to be performed on the results of a query. Specifically, the transformation process transforms the result tiles into transformed versions as query results are generated. In one embodiment, the transform performs an image processing morphological operation that dilates or erodes image features in the result tiles to show spatial support. In one embodiment, the conversion process is a deconvolution process that separates colors corresponding to the stains used in preparing the tissue to be imaged. In one embodiment, a tissue quantification process is performed that identifies and localizes tissues such as cell nuclei, stroma, glands, and the like. In one embodiment, the localization and identification of such structures is used to transform the result tiles and amplify the targeted cellular structures.

In one embodiment, the query results are passed to a defined analysis process that generates at least one transformed result for each returned result as it is generated incrementally. In one embodiment, this process retains the transformed results for processing and query operations. By holding in this way, the tiles can be used in the same way that the original tiles were used. For example, at least one transformed tile can be designated as a query tile, and each time a transformed result is generated, a query operation is performed over that transformed result. These transformed results were generated from the original query operation on the original tiles. In one embodiment, the operations of relevance feedback and/or adding query tiles can be performed during query execution.

In one embodiment, combinations of query and analytical transforms are chained together, where the output of a query process becomes the input of a transform process, and the output of that transform process becomes the input of another query process. In one embodiment, this chain of query and analysis processing is virtually unlimited. In such a processing chain, for example, the process begins with the user selecting some query tiles from the base layer of the image. The user then selects the slides targeted by the query, then the index to be used for the query (e.g., a color histogram-based index) is chosen, and the query is executed when the user chooses to execute. , and the user specifies that the analysis should be performed on the results and that the analysis should perform color deconvolution on the result tiles. is converted to another tile (eg, H&E (Hematoxylin & Eosin) stained tile) and the conversion result is generated and given to the user. Each time this query process produces additional results, those results are automatically converted and presented to the user. The user then selects one of the transformed tiles (eg, the one corresponding to the hematoxylin stain) as the query tile. An index for this query is chosen based on texture feature extraction. When this query is run, results are determined from the multiple transformed hematoxylin tiles. The process returns more results as each step in the chain above produces more results.

In one embodiment, embodiments of the processes described herein are performed when a new slide is added to the image repository. In one embodiment of this case, the final result similarity measure is thresholded, and results exceeding the alert threshold are sent to the alert generation system. In one embodiment, the addition of a slide to the repository triggers a processing pipeline, and an alert notifies the user of the new slide and the result set tiles associated with that pipeline for evaluation. This is an automated screening process for scanned slides that utilizes existing processing pipelines to automatically process slides added to the system and generate notifications or alerts for further processing. Such automated processing can be applied to screen slides for a variety of purposes, including abnormal tissue detection and quality assurance.

In one embodiment, the query tile is specified from the source slide. A scanned specimen slide is stored in an arrangement that tiles a series of different scales. The highest scale is taken as the original magnification at which the slide was imaged. Typically, this magnification is an optical magnification of 40x (hereinafter "x" may be described as "x (multiplicative operator)"). The high scale tiles are subsampled to lower scales, each lower scale being 1/2 the scale factor of the previous scale. For example, if the highest scale is 40x, then the next highest scale is 20x, then 10x, 5x, 2.5x, 1.25x, 0.62x, 0.31x. For example, at the lowest scale, a typical tissue sample occupies 4-8 tiles. Each tile has 256x256 pels. In one embodiment, a target slide is specified on which to perform a search for matches with the query tile. For example, the order in which a target tile is compared to a query tile determines the exhaustive traversal of target tiles at the same scale as the query tile using a comparison operator based on the L2 norm of the pels between the comparing tiles. Ordered. In one embodiment, a multi-scale search and comparison is performed in which at least one available low-scaling scale is used to generate match hypotheses, and then increasing the scale until a target magnification is reached to prove the match hypotheses. .

The magnification is 1/2 for each successive scale, and the number of pels in each dimension is also 1/2. Thus, the collection of tiles at each scale is a sub-sampling of the previous higher magnification scale. One tile at one scale corresponds to four tiles at the next higher magnification. This correspondence matches the quadtree decomposition of the original full-scale slide image at the highest magnification. The present invention traverses this quadtree where each tile is a node of the tree and each spatial correspondence is an edge of the tree.

In one embodiment, such a quadtree traversal for multi-scale search matches spatially corresponding tiles at a low scale factor to the presence of match candidates at a high scale factor up to the query tile scale factor. used as an indication of In one embodiment, foundation traversal prioritizes further comparisons between subtrees of the lowest scaled tiles by comparing the corresponding tiles at the lowest scaled to the query tile. In one embodiment, the lowest magnification level tiles are ordered based on a similarity measure. In one embodiment, at least one tile on the side with the lowest similarity is excluded based on its similarity falling below a given threshold. In one embodiment, this threshold is 0.70 correlation of feature vectors derived from tiles. In one embodiment, the feature vector for each tile is a 32-bin histogram based on the sum of the spectral components of that respective tile. For each retained tile, the corresponding four tiles at the next higher magnification are added to the new collection of tiles. This collection of tiles is evaluated based on the same process as the previous level, and this recursive process continues until the base multiplier is reached. In one embodiment, the correlation threshold starts at a lower value of 0.50 and increases incrementally up to a maximum of 0.80 for each magnification level. The embodiment described here is a breadth-first traversal for a quadtree, comparing the corresponding query tile with a low scale factor to its corresponding search tile at its current level, then the next higher level. Move to Magnification. This is equivalent to recursively processing the next level based on all matches at the current level in a single batch. In one embodiment, such recursive processing is performed in batches that are evenly divided from this single batch. Each of these multiple smaller batches is also recursively processed to higher magnification levels. In embodiments where the batch size is one tile at the current level, the above quadtree traversal becomes a depth-first quadtree traversal. When the batch size per level is variable between a single batch (breadth-first) and one tile per batch (depth-first), it is called adaptive traversal.

In one embodiment, adaptive traversal is set to breadth-first at the start of the search. As the search progresses, the computational cost is calculated as the number of similarity computations performed. In terms of computational cost, for example, the number of search results returned per comparison determines the incremental search result latency. As an example, if the number of similarity comparisons is 200 and the number of results returned with a similarity measure of 0.90 for correlation is 40, then the comparison:result ratio is 200:40, That is, there are five result latencies per result. For such situations, it is determined that breadth-first partitioning, which partitions all tiles at the current level into a single batch, is efficient. If the resulting latency increases by an amount above a given threshold (e.g. 50), this signals the adaptive traversal mechanism to increase the number of batches per level (e.g. two). . If the resulting latency is still above the threshold, the number of batches for each level is increased to three. The threshold can be set to any number desired or required by an administrator or system. In one embodiment, the adaptive mechanism sacrifices processing efficiency for each level in order to target results with high similarity before moving on to results with low similarity (depth-first processing). . At some point, the number of batches may equal the number of tiles at the current level, which corresponds to strict depth-first traversal for quadtrees.

In one embodiment, the adaptive traversal incrementally divides the batch process to increase the number of batches per level in order to reduce the result latency. For example, if a process reaches its depth-first batch split limit but results latency increases, the process tapers down the number of splits for searching to reduce result latency. In one embodiment, a computational budget is specified to determine the number of subtrees to be evaluated.

In one embodiment, a sampling function is provided for accessing data (eg, tissue image data). This sampling function defines the order in which data is accessed. This facility builds an access plan based on constraints predicted from the data itself and user interaction. These constraints on sampling functions constitute the query context according to the invention.

In one embodiment, sampling consists of access plans based on user-specified and system-specified. The user specifications include the scope of data to be retrieved in addition to any predicate specifications. The system specification includes existing data and remaining results from past processing. The sampling function incrementally returns a set of partial results. The results returned in this manner are used to modify the user-specified and the system-specified within the current query context. A refinement operation interrupts the processing so that the query is directed to a more relevant result set (eg, sample).

FIG. 3 is a block diagram illustrating query unit 300 in an exemplary embodiment. This query unit 300 includes a domain (system) specification 302 and a user specification 310 for the query. Domain specifications 302 are combined with user specifications 310 to form predicates and/or targets 304 that produce results 306 . The query is refined and/or expanded 308 based on the results to change the user specification 310 of the query.

FIG. 4 is a block diagram illustrating analysis unit 400 in an exemplary embodiment. Analysis unit 400 generates transformation results 406, 408 based on data returned from queries and/or user specifications 410 of the repository. The transformed data is in the form of an image of the same dimensions and resolution as the data from which it was derived, with variations in individual pel positions and color depths based on the transformation. A domain specification 402 examines the results and determines and/or transforms the results.

FIG. 5 is a diagram illustrating the architecture of query unit 500 at an intermediate stage in an exemplary embodiment. An intermediate result (“intermediate”) is the result of a defined process performed to produce a result. Said intermediate depends on a combination of the underlying state of the repository and a defined process.

In one embodiment, a region of interest (“ROI”) 502 defines at least one adjacent pel in a repository that constitutes one tile 512 in that repository. ROI 502 may be a complex polygon 522 . Such a polygon 522 is defined in at least one tile, and the interior of the polygon 522 is interpolated to discrete locations on one or more tiles. The data for a tile is the scale and spatial decimation of the image data and forms the basic unit of processing for that tile. Vector 514 is the feature vector 504, 516 extracted from the tile and used to generate the index against which the query operates. Correlation intermediate 506 calculates the correlation between two tiles based on the similarity between the extracted feature vectors of the two tiles (e.g., one of the two tiles is the query predicate and the (similarity generated when the other of them is a query result). The correlation itself is the feature index that is searched. A tensor 526 is constructed to convey the correlation to other feature vectors. Layers 508 are any information derived from the data after transformation (518) and/or filtering and/or visualization. Quantification 524 is a quantitative incremental process involving, for example, aggregate tile processing (eg achieved by scale-based constraints). Measures 528 are any of scale-based constraints, thresholds, correlations, indices, and other information.

In one embodiment, the structure and content of the data define the order of processing priority, and this order of priority determines the order in which the data is presented to the user. This allows the user to understand both the structure and content of the data. The user is guided in the search for data by this order of preference. This alleviates the need for prior knowledge of the data.

In one embodiment, the processing is tailored to the returned data and expands the sampling range of the retrieved data based on returning a sparse set of results. Similarly, in one embodiment, the sampling range is limited when a large amount of data is returned. Such restrictions allow the data to be extensively sampled while preventing a large amount of data from being returned for a small, localized subset of the total data.

In one embodiment, spatial continuity yields the assumption that spatially adjacent data are generally more related than distant data. Similarly, when the correlation between nearby data and distant data is large, the correlation between nearby data and adjacent data also becomes large for the same reason. In one embodiment, these relationships are used as a basis for predicting search constraints.

In one embodiment, usage statistics are used to extend the processing scope of the data generated and accessed to a wider range than other data. Intermediate data restriction and flush operations are performed when data is generated and accessed infrequently. The frequency of access influences the ranking of data. IAPE (ranking data for user exploration) adjusted for quality assurance (“QA”) biases compared to system processing-based rankings (e.g., biases based on slide screening, etc.) be done.

In one embodiment, the system and method provide a means to expand and/or restrict the results when they are returned to the user. This online reconciliation facility provides users with a means to interact with the query results. This functionality is made available to the user at any point in the query execution process (even after the query has finished). In such cases, the query is re-executed with the new constraints. In one embodiment, the query and the incremental results of the query are analyzed to determine if a given limit has been reached that makes further processing of the query inefficient. If it is determined that the limit has been reached, the query may be stopped and the user may be given an opportunity to make changes to the query.

In one embodiment, data partitioning and traversal is performed to optimally maintain storage and computational associativity. In one embodiment, the data is regularly divided into non-overlapping spatial regions (eg, blocks, tiles, etc.). In one embodiment, the data is subsampled and split. In one embodiment, the divided data tiles are ordered based on scale and spatial proximity. The locality of each individual tile serves as the basic unit of computation. This elementary unit is processed and the result is given to the user.

In one embodiment, aggregate tile processing is achieved primarily by scale-based constraints that use analysis on low scales to test the suitability of data processing order at high scales. For example, the scale-based pyramidal structure of tissue image data is represented as a quadtree decomposition of that image data. The relevance of this representation is checked using parent-node affinities. Increased access to sidetrack information yields a large number of incremental results. This means that only part of the pipeline needs to be executed.

In one embodiment, the processing of the algorithm depends on the amount of results returned. The traversal policy determines how the traversal strategy is changed.

In one embodiment, where the organization of image data storage and the organization of its derived data can be expressed by a quadtree decomposition, the granularity for each processing increment is based on the processing of subtrees of that quadtree. The operations required to return the result set are bounded using the subtree processing cost estimates. Separation of processing into subtrees facilitates the application of distributed parallel processing to scale the operation of traversal.

In one embodiment, a traversal for a quadtree is defined to process subsets at each level, and the number of subsets at each level determines the degree to which the traversal is breadth-first or depth-first. to decide. A breadth-first bias can sample a large amount of data and uses a large amount of computational resources to return a set of results. Such an incremental approach is advantageous when the matching results are sparse and scale-related weakly. In one embodiment, breadth-first traversal is generally exhaustive and does not make many assumptions about the distribution of matching samples. This suggests that the predicate retrieval hypothesis is weak. In one embodiment, depth-first bias samples a smaller amount of data resulting in shorter computation times and returns results in smaller increments. This is advantageous when the results are dense and highly scale-related.

In one embodiment, searching the data produces a number of partial solutions for subsequent queries. These partial solutions represent an opportunity to provide results more quickly than those calculated from less intermediate data. A quality bias exists where many of these partial results were generated by the activities of other users. Also, the subset of results with this bias is returned.

In one embodiment, this quality bias is not only available to improve the efficiency of returning results, but is also recursively referenced on a data-by-data basis and as an overall ranking of data usefulness (e.g. counted ).

In one embodiment, tissue image data challenges are addressed by system flexibility that allows for user-specified refinement in addition to downstream processing in the pipeline. This specification refinement is used to modify the current query processing to change the results produced by that query. Downstream processing operates on the query results as they are generated to perform additional transformations on the data. Additional query processing is then performed. Query processing flexibility can provide users with the flexibility to explore data through query modification and further processing.

The underlying image data is structured and organized for incremental processing across spatial and spectral scales. Data are spatially and spectrally normalized by a calibration process upon import. Correlation of data is determined by similarity of feature indices and maintained in a correlation index. Access and processing of data are estimated, and such access processing is performed based on predicted and actual costs. Preliminary computations are performed to approximate the results of the overall computation to provide computational cost estimates and to incrementally compute the final results. The results are rolled up and aggregated for future operations, and the intermediate products are retained for update operations that allow the algorithm to operate online.

In one embodiment, the kernel utilizes a pyramidal/hierarchical/quadtree data structure (e.g., multi-scale image pyramidal structure) to facilitate incremental and discrete operations. do. In one non-limiting embodiment, a tile is a square image with a spatial extent of 256 in each dimension. These tiles were generated from the original image by successive 2x subsamplings of the original image until the dimensions of the recursively subsampled image were less than 256. The file system directory containing these tiles is also divided into 256 groups or arrangements of 16x16 tiles. This file system organization provides an optimal arrangement for storage system locality to take advantage of caching mechanisms. Furthermore, such an isolated configuration makes it possible to envisage a distributed file system in which sub-region processing can be executed without the need to share context information between separate computing environments. .

In one embodiment, feature vectors based on histogram bins are utilized in tile-to-tile similarity comparisons. Such a feature vector represents an approximation of the contents of the tile that is used for index generation. For tiles that are similar to each other in such spectral feature space, the processing on these tiles is used to estimate the results of processing involving the more computationally intensive feature extraction process.

In one embodiment, the scale aspect of our approach emphasizes the spectral feature vector, which is an approximation to the spectral content of the tile, to be a ranked approximation. For example, if the feature vectors are the size of a tile, the location of each feature vector corresponds directly to each pixel. In one embodiment, when there are fewer bins than pixels, it necessarily means that the original data has been scaled down. Spatial invariance is implied when histogram bins and pixel locations no longer correspond.

In one embodiment, the kernel operates online, performing coarse-grained operations and aggregating the results. The computational cost of performing those operations is factored in the processing scheduling operations. A cost estimate is performed on the subtrees of the access plan and they are aggregated. These estimates allow adjustments to the allowed operations on each subtree.

In one embodiment, the kernel is configured to act incrementally on the pipeline of processing elements. These elements may be such as a query element followed by an analysis element. The query element applies at least one predicate pattern against a set of candidate patterns and returns results based on pattern matching conditions. Analysis is performed on the result set pattern to somehow transform the result set pattern for visualization, further analysis and/or for querying operations.

In one embodiment, intermediate products are generated based on the query itself generating indexes, and these indexes are used in the pattern matching process. In one embodiment, the analysis process produces intermediate products in the form of filtered or transformed input image data or quantitative measures.

In one embodiment, other intermediate products also include intermediate products from the approximation function that produce post-approximation results. Also, other intermediate products from on-line operations are retained for the purpose of increasing iteration speed. Intermediates from such online operations are also considered pipeline intermediates.

In one embodiment, the retention and utilization of these intermediates allows the kernel to take an alternative approach to producing results without the operations required to repeat these processes.

In one embodiment, data such as tissue image data have different structures at different scales. These structures are not necessarily structurally related across different scales. That is, the structural pattern is not necessarily repeated. The relationships between these patterns may be modeled as generative functions that allow macroscale models to generate at least one microscale model. These models can be used to impose additional constraints on queries. A kernel-specific aspect of these models is that the kernel finds such macro/micro-models regardless of the specific data, and that the kernel utilizes the macro-/micro-models to collaborate across different scales. Providing similarity.

In one embodiment, regarding the data management aspect, the usefulness of the system depends on the nature and organization of the data. For tissue image data applications, retention of original image data is prioritized over retention of derived data. The data management system configuration reflects and references archiving priorities to define storage, transfer and analysis operations.

In one embodiment, the size of the data imposes a practical limit on data replication operations. Given this size, along with the long-term retention policy associated with that data, it is possible to meet the requirement by building a database around archived data. The choice of archive format and data layout affects the functionality of all downstream processing.

In one embodiment, data storage is based on the ability to manage multiple different layers of data derived from the data. This data retention and flushing operations are performed to meet storage and computational requirements based on the ability to regenerate such data on demand.

In one embodiment, the processing associated with transferring and distributing data is accomplished through physical groupings of data and the ability to make references outside the scope resolved by a given addressing scheme. By imposing restrictions on dependencies, the system provides operational advantages when utilizing processing in a distributed environment.

In one embodiment, the system has actions that are automatically performed based on both routine processing and user interaction.

Figures 7A and 7B are a simplified flow diagram illustrating a method of initiating a query to retrieve data in an exemplary embodiment. As shown in FIG. 7A, after the search is initiated (block 705), the current zoom level (or "magnification", used interchangeably) of the image query image is greater than a predetermined threshold Th1 (block 710) If so, search results for the current level of the query are returned (block 715).

In one embodiment, the threshold is set to limit unproductive search operations, eg, to terminate the search if the number of search matches generated is not high. As another example, the threshold is set to limit unproductive search operations before image pixels no longer contain meaningful image information. Alternatively, the threshold is set to limit the depth of magnification levels or other depth in the search. For example, if a subset is too small to generate a small number of matches from that subset, then the size of the subset is increased for a broader search.

In one embodiment, if the search result was pre-computed or was computed during a previous search, the search result is returned without performing a depth-based search.

In one embodiment, the search function is recursively initiated with different thresholds.

In one embodiment, each zoom level or magnification corresponds to one level in the quadtree. For example, the image consists of a single tile at magnification level one. Then, the correspondence between that image and its four children is considered to be at magnification level 2, the 16 tiles that are children of these four children are considered to be at zoom level 3, and these 16 children are considered to be at zoom level 3. … and so on. Each magnification level has a higher resolution than the previous magnification level, typically doubling the resolution in each dimension. If the tiles are at the resolution limit, the children for those tiles are equivalent only if the pixels are interpolated. If such interpolated pixels are also utilized, these are considered to be the maximum zoom level or magnification. Decomposition into quadtrees results in a parent tile being a down-sampling, low-pass spatial filtering of its four corresponding child tiles, so that the child tiles are related to their parent tiles. For example, if a given color is found in a parent tile, the same color is likely to be found in a child tile, or at least the parent side color can be derived from the child tile side color by a downsampling process. be.

In one embodiment, to retrieve search results, a list of result tiles that meet the current search criteria is queried from memory. In one embodiment, such a list of tiles includes associated data (e.g., tile level and/or query and/or index filename and/or result size and/or index associated with each tile). index type).

In one embodiment, the current search criteria or query limits search results based on priority and system resources. For example, the query may include limits on the operations available for the search, limits on the time available to complete the search, limits on the amount of memory available, thresholds on the quality or quantity of search results, etc. .

In one embodiment, if the current magnification level is not greater than a predetermined threshold Th1 (block 710), the next level tile is retrieved (block 720). This retrieves or creates the next level tile quadtree child that corresponds to the current level. Then, from those next level tiles, a list of tiles that match the current query result is generated (block 725). In one embodiment, the query is refined each time more results are found. For example, the query contains a minimum result size. Then, for example, when a better match is found, the broader result or a given result found earlier is removed from the result list by narrowing the query that retrieves the result from memory.

In one embodiment, for each tile in the retrieved list, that tile is added to the subset (block 730). If the size of the subset, the number of tiles in the subset is greater than or equal to a predetermined threshold (block 735), then a recursive depth-first search is performed on the tiles in the subset (block 740). This reduces the size of the set on which the recursive search is performed. The results of this recursive search are saved (block 745) and the set is cleared (block 750). A recursive search is then performed on the remaining set (block 755), the results are saved (block 760), and the set is cleared (block 765). The saved results are retrievable from memory from the time the results were stored. This allows, in one embodiment, to utilize a continuously updated set of search results, both during execution of the search and after the search has finished.

Figures 8A and 8B are a simplified flow diagram illustrating a method of performing a recursive search in one exemplary embodiment. As shown in Figure 8A, a recursive search is initiated for a set of tiles (block 805). To optimize the search, a new query tile is formulated (not shown), eg, as the first child tile for the first tile in the input tileset. Then, for each tile in the tileset, the next level tile is retrieved (block 810) and the next level tile is added to the result set (block 815).

In one embodiment, once the result set is registered, if the current zoom level is the target zoom level (block 820), the quality of matches within the result set is evaluated (block 825). For example, if the first tile in the result set has a match value of less than 50% compared to the query tile, the results are too different from the query tile, and depth searching may Do not continue along this edge (branch) of the quadtree (block 830). On the other hand, if the results are sufficiently precise, the current result set is returned as the result of the recursive search (block 835). A minimum quality threshold is set as part of the query.

In one embodiment, match quality is evaluated as the difference between the vectors. For example, each pixel in a tile has multiple numerical values representing the color and/or brightness of that pixel. Each tile then has an array or vector of such numbers for every pixel in that tile. The two tiles are then compared by calculating the distance or mean squared error between the vectors of each tile.

In one embodiment, if the current zoom level is not the target level (block 820), the recursive search continues as shown in Figure 8B. As shown in FIG. 8B, for each tile in the result set, if the size of the subset is less than a predetermined threshold Th3 (block 840), that tile is added to the subset (block 845). On the other hand, if the size of the subset is greater than or equal to the predetermined threshold Th3 (block 840), a new recursive search is initiated for that subset (block 850). The results of this recursive search are added to the interim result set (block 855) and that subset is cleared (block 860). This reduces the size of the set on which the recursive search is performed, as described above. After each tile in the original result set has been processed and these tiles have been added to the subset, a recursive search is performed on the remaining subset (block 865) and the results are added to the interim result set. (Block 870) and the subset is cleared (Block 875). The provisional result set is then returned as the result of this recursive search (block 880).

In one embodiment, the list of one or more tiles at each level is sorted based on the query tiles that match them, or at least one of at least one of the query tiles if not at the target level. Sorted based on degree of matching with parent tiles.

FIG. 9 is a diagram showing the layout of slides in one embodiment of the present invention. Specifically, a biological tissue or other specimen is placed on a query slide. From this query slide a query slide image is prepared. For example, this query slide image is a digital file that has been spatially segmented into square or other shaped tiles. From these query slide tiles, at least one tile is identified as at least one query tile (or search tile). The information obtained from the query tiles is then input into the similarity measure engine. In one embodiment, the similarity measure engine normalizes the query tiles to a desired size and memory value.

In one embodiment, normalization may include any available method. In one embodiment, normalizing at least one tile or slide image includes obtaining metadata information regarding micron storage pixel values. For example, if image A is 20 microns per pixel scale and image B is 40 microns per pixel scale, the intermediate level is to equalize the microns per pixel scale for these two slides (e.g. , both are calculated as 20 microns per pixel scale, or other levels, etc.). In one embodiment, one may look for the highest resolution capture available, medium resolution capture available, or lowest resolution capture available to yield different results depending on the desired image search. . In one embodiment, color normalization or color correction is performed. For example, the brightness, intensity and color of at least two tiles or images are determined and changed to similar levels for retrieval. For example, if the same slide is processed on two different machines (or if two different scanning operations are performed), the two resulting images can be color-normalized or color-corrected to achieve the same result. Any other slides from these machines can be corrected based on the results of the determination. For example, the luminance values, RGB values, and other light-based or color-based parameters of two scanned images are compared and a decision is made to change one or both to a particular set of parameter levels, and the same source The same changes or corrections are made with respect to color (color, brightness, brightness, etc.) for subsequent images from .

In one embodiment, the similarity measure engine compares at least one query tile with target tiles located at one or more locations. For example, target tiles are located in one or more databases at one or more physical locations or servers. For example, at least one of the target tiles is from one or more target slide tiles. The one or more target slide tiles are prepared from the target slide image or the target slide digital image. The target slide images are uploaded from a source and/or generated from at least one target slide. The target slide was prepared with a tissue specimen or other sample.

FIG. 10 is a diagram illustrating an example of tile feature comparison. A query tile or digital image query is input to the feature extraction engine. The feature extraction engine determines certain parameters and/or features of the query tile. For example, the feature extraction engine uses a predetermined set of features to generate data and/or measurements on color pixels, size, and/or the like. Some or all of the data and/or measurements determined using the feature extraction engine are input to a comparison engine, which compares at least one target tile of at least one feature of the query tile. data and/or measurements for similarity. The data and/or measurements for the at least one target tile were obtained using a feature extraction engine or the like.

FIG. 11 is a diagram illustrating an example of spatial decomposition of tiles in one embodiment of the present invention. A slide image is a digital file or other electronic data or information that can be identified as a slide tile or piece of the slide image. The slide tile is then decomposed into lower level sets of tiles.

FIG. 12 is a diagram illustrating an example of scale decomposition of tiles in one embodiment of the present invention. A slide image is segmented into tiles and one of the tiles is decomposed into lower level tiles. One of these lower level tiles is then used to search for other similar tile images using, for example, various embodiments described herein.

The present invention, including any embodiment described herein, controls or executes data processing apparatus such as digital electronic circuits, computer hardware, firmware, software, computer program products, machine-readable storage devices, processors, computers or the like. can be realized with a propagating signal that The invention, including the embodiments described herein, can be written in any form of programming language and can be implemented as a stand-alone program or as a component of another program. A computer program may be deployed and/or stored and/or executed and/or transmitted to and/or transmitted to one or more computers at a single location or at two or more locations. can be sent from any computer.

In the present invention, any step (step) of the method is performed by at least one programmable processor, computer, tablet, smart phone, portable smart device, etc., executing a computer program to operate on input data and generate output. It is done by doing as you do. Storage media include EPROM, EEPROM, flash memory devices, chip cards, magnetic cards, barcodes, QR codes (registered trademark), DVD-ROMs, CD-ROMs, internal hard disks, removable disks, magnetic disks, magneto-optical disks, Optical discs and the like are included. The present invention enables user interaction by displaying from a computer to a display device, cathode ray tube (CRT) monitor, liquid crystal display (LCD) monitor, LED monitor, touch screen, or the like.

The present invention processes large amounts of data in some implementations. Data is stored in back-end components such as data servers, application servers, cloud servers, and user interface functions such as keyboards, voice input keyboards, graphical user interfaces, web browsers, and the like. In the present invention, the components are interconnected by any form of digital data communication, such as a communication network such as a local area network (LAN), wide area network (WAN) (eg, the Internet).

The foregoing description and illustration of the embodiments should be construed as illustrative only and not limiting of the present invention. For example, different features/components included in different embodiments of the embodiments described above may be implemented in various combinations, such as with or without each other. Various alterations, modifications and improvements are possible in light of the above teachings and the scope of the appended claims and are intended to be within the spirit and scope of the invention.

Although the invention has been described with reference to specific examples and specific embodiments, it is to be understood that the invention is not limited to these examples or embodiments. The invention also encompasses variations on the specific examples and embodiments described herein.
In addition, this invention includes the following contents as a mode of implementation.
[Aspect 1]
A computer-implemented method for query-based searching of a database of images comprising:
returning a first list of result tiles satisfying the query at that query magnification level in response to determining that the query magnification level is greater than a first threshold;
retrieving a tile with the next lower magnification level in response to determining that the query's magnification level is less than or equal to the first threshold and results in a second list that satisfies the query at the next lower magnification level. the process of returning the tiles;
processing the result tiles of each list, wherein for each result tile:
adding the resulting tile to the subset of resulting tiles;
recursively searching the subset in response to determining that the total number of result tiles in the subset is greater than or equal to a second threshold;
saving the results of each recursive search for said subset into a remaining subset;
recursively searching the remaining subset; and storing results of searching for the remaining subset;
a process comprising
A method, including
[Aspect 2]
A method according to aspect 1, wherein each magnification level corresponds to a level in a quadtree, and each level in the quadtree includes tiles representing image results.
[Aspect 3]
3. The method of aspect 2, wherein a child tile has an association with a parent tile.
[Aspect 4]
4. The method of aspect 3, wherein the parent tile is at least one of downsampling and low-pass spatial filtering of the corresponding at least one child tile.
[Aspect 5]
3. The method of claim 2, wherein the step of retrieving the next lower magnification level tile is generating children at the next lower level in the quadtree.
[Aspect 6]
2. The method of claim 1, wherein the query includes at least one of a minimum threshold number of results and a maximum threshold number of results.
[Aspect 7]
2. The method of aspect 1, wherein the query includes an image, and the magnification level of the query is the magnification level of that image.
[Aspect 8]
8. The method of claim 7, wherein a result tile is at least one of a scale factor of the result tile, an image of the query, a filename of an index associated with the result tile, a result size, and an index type. Based on the method to be included in the returned list.
[Aspect 9]
2. The method of aspect 1, wherein the first predetermined threshold is defined such that the method terminates in response to determining that the number of search results falls below a certain number.
[Aspect 10]
2. The method of aspect 1, wherein the query is updated after returning the first list of result tiles.
[Aspect 11]
The method of aspect 1, further comprising:
excluding results from at least one of the first list of result tiles and the second list of result tiles before processing each list of result tiles;
A method, including
[Aspect 12]
2. The method of aspect 1, wherein the query includes a threshold level of quality.
[Aspect 13]
2. The method of aspect 1, wherein the query includes a time limit, and the search is performed with the time limit.
[Aspect 14]
A method for performing a recursive search of tilesets based on query tiles, comprising:
For each tile in said tileset, until the result set is full:
retrieving a set of tiles from a next level; and adding said next level tile set to said result set;
and
evaluating the quality of matches in the result set in response to determining that a magnification level is at a predetermined target level;
In response to determining that a magnification level is below the target level, for each tile in the result set:
adding the tile to the subset in response to determining that the number of tiles in the subset is greater than or equal to a third threshold;
In response to determining that the number of tiles in the subset is less than the third threshold,
recursively searching the subset;
adding the results of this recursive search to a provisional result set;
clearing the subset;
to run
recursively searching the subset;
adding results of a search to the preliminary result set;
clearing the subset and returning the interim result set;
and
A method, including
[Aspect 15]
15. The method of aspect 14, wherein assessing match quality comprises determining whether a match value of a first tile in the result set compared to the query tile is less than a predetermined value. A method having to.
[Aspect 16]
15. The method of aspect 14, wherein each pixel in a tile has at least one numeric value representing at least one of color and brightness of each pixel, and each tile has A method comprising a vector of said at least one numerical value for every pixel.
[Aspect 17]
A computer implemented method for continuously processing a repository of image data comprising:
receiving a query specification containing a request for data;
receiving a system designation of the computer on which the method is to be performed;
comparing the query specification and the system specification to determine a domain specification;
initiating queries to the repository based on the domain specification;
receiving results of the query, including image data;
displaying an interactive and iterative exploration of the resulting image data in a graphical user interface;
receiving input of the resulting image data via the graphical user interface;
updating the query based on the input received;
displaying the updated graphical user interface based on updated result image data;
A method, including
[Aspect 18]
18. The method of aspect 17, wherein the repository of image data comprises microscopy digital data.
[Aspect 19]
18. The method of aspect 17, wherein the successive processing of the image data produces results incrementally such that the results are available before the processing is completely finished.
[Aspect 20]
18. The method of aspect 17, wherein the query specification implicitly defines data indices and transformations.

Claims (14)

1. A computer-implemented method for searching a database of images based on one or more query tiles included in a query, comprising:
retrieving the original capture resolution of an image in the database of images using a query tile for comparison and providing this query tile;
retrieving an image stored in the database having a magnification level equal to or greater than the query tile magnification level in response to determining that the query tile has a magnification level greater than a first threshold; returning a first list of result tiles that satisfy the query tile at that magnification level;
In response to determining that the query tile's magnification level is less than or equal to the first threshold, an image stored in the database is searched for a tile with the next lowest magnification level to the query tile's magnification level. retrieving and returning a second list of result tiles that satisfy the query tile at the next lowest magnification level;
processing the result tiles of each list, wherein for each result tile:
adding the resulting tile to a subset of the plurality of subsets of resulting tiles;
recursively searching the subset for the query tiles in response to determining that the total number of result tiles in the subset of result tiles is greater than or equal to a second threshold;
saving results of each recursive search for a subset of the result tiles into a remaining subset of the plurality of subsets;
recursively searching the remaining subset for the query tiles; and saving results of searching for the remaining subset;
a process comprising
A method, including 2. The method of claim 1, wherein each magnification level corresponds to a level in a quadtree, and each level in the quadtree contains result tiles representing an image. 3. The method of claim 2, wherein child tiles have relationships with parent tiles. 4. The method of claim 3, wherein the parent tile is at least one of a downsampled tile and a low-pass spatially filtered tile of the corresponding at least one child tile. 2. The method of claim 1, in response to determining that the total number of result tiles in the subset of result tiles is less than the second threshold,
recursively searching a subset of the result tiles for the query tile;
Add the results of the recursive search to the provisional result set,
Clear said subset, then
recursively searching the remaining subset for the query tile;
adding the results of the search to the interim result set;
clearing said remaining subset;
A method that returns the interim result set. 2. The method of claim 1, wherein the query includes an image and the magnification level of the query is the magnification level of that image. 7. The method of claim 6, wherein a result tile is at least one of the result tile scale, the query image, the index filename associated with the result tile, the size of the result, and the type of index. The method to be included in the returned list based on. 2. The method of claim 1, wherein the second predetermined threshold is defined such that the method terminates in response to determining that the number of search results falls below a certain number. 7. The method of claim 6, wherein the query is updated after returning a preliminary result set. 2. The method of claim 1, wherein the query includes a threshold level of quality. 2. The method of claim 1, wherein the query includes a time limit, and the search is performed with the time limit. A computer-implemented method for performing a recursive search of a tileset based on a query tile, comprising:
For each tile of the current tileset in the recursive search, until the result set for said query tile is full:
retrieving a set of tiles from the next level of the current tileset; and adding the next level tileset to the result set;
and
evaluating a degree of match of the result set to the query tile in response to determining that the query tile's magnification level is at a predetermined target level;
In response to determining that the query tile's magnification level is below the target level, for each tile in the result set:
adding the tile to the subset in response to determining that the number of tiles in the subset is greater than or equal to a third threshold;
In response to determining that the number of tiles in the subset is less than the third threshold,
recursively searching the subset for the query tile;
adding the results of this recursive search to a provisional result set;
clearing the subset;
to run
recursively searching the subset for the query tile;
adding results of a search to the preliminary result set;
clearing the subset and returning the interim result set;
and
A method, including 13. The method of claim 12, wherein the step of evaluating a degree of match to the query tile comprises: if a first tile in the result set has a match value compared to the query tile less than a predetermined value; A method comprising determining whether there is a 13. The method of claim 12, wherein each pixel in a tile has at least one numerical value representing at least one of color and brightness of each pixel, and each tile has a vector of said at least one numerical value for every pixel in .

Images