JP2021114305A - Continuous image processing - Google Patents

Abstract

To provide a method for querying, analyzing and processing data.SOLUTION: A method for searching a database of images based on a query comprises: if a magnification level of the query is greater than a first threshold, returning a first list of result tiles satisfying the query at the magnification level of the query; if the magnification level is less than or equal to the first threshold, retrieving tiles at a next lower magnification level and returning a second list of result tiles satisfying the query at the next lower magnification level; and processing each list of result tiles, the processing including, for each result tile: adding the result tile to a subset of result tiles; if the number of result tiles in the subset is greater than or equal to a second threshold, recursively searching the subset; saving results of each recursive search of the subset in a remaining subset; recursively searching the remaining subset; and saving results of the search of the remaining subset.SELECTED DRAWING: Figure 1

Description

Related application

This application is the priority of US Provisional Patent Application No. 61 / 905,027, referred to as “Continuous Image Analytics” filed November 15, 2013, and “Continuous Image” filed November 15, 2014. Claims the priority of PCT International Application No. PCT / US14 / 65850 called "Analytics (Continuous Image Processing)". The entire teachings of each application are incorporated herein by reference.

Copyright reservation

A part of the disclosure contents of the documents of the present application includes works subject to copyright protection. The copyright holder does not object to the reproduction of the documents of the present application or the disclosure contents of the present application displayed on the patent bag or record of the Japan Patent Office, but reserves all copyrights otherwise. do.

It provides a set of computer-readable instructions in methods, systems and storage media (eg, non-transient storage media) that query, analyze and process data. Specifically, methods, systems and storage media (eg, non-transient) that process samples used in digital systems, query a database of images, iteratively process image data, and generate image data results. Provides a set of computer-readable instructions in a storage medium).

Relevant data can be retrieved from the data repository by requesting it with a query. However, in existing database-query systems, there is a semantic gap between the user's conceptual expectations (eg, the conceptual expectations conveyed by the query) and the lower-order representation of the data. Specifically, in the case of tissue imaging (tissue imaging), the semantic assignment of meaning between the lower representation of a digital image of microscopic tissue and the user's intent (ie, the intent conveyed by the query). There is a gap. The size of this semantic gap is so large that it makes it impossible to apply content-based image retrieval techniques universally. Specifically, if the query is too broad, irrelevant results may be returned, and if the query is too specific, relevant results may be eliminated.

The problem of semantic gaps is due to the complexity, variability and size of the data. These factors complicate the algorithm's distinguishing means (eg, act contrary to the distinguishing means), and in extreme cases, when the algorithm is extended beyond the scope of the constraint, the error model becomes a pattern of image data. It can dominate the model. Another problem is the problem of practical approximation assumptions used when applying algorithms to large amounts of image data. The approximation here refers to a subset or summary of image data to make the algorithm easier to handle computationally. In the case of tissue image data, the scale of the data is extremely wide, which not only makes the distinguishing features esoteric, but also makes the distinctive features different in meaning on different scales.

The present invention is a method of searching a database of images based on a query, which is executed by a computer, in response to a determination that the magnification level of the query is greater than a first threshold. In response to the process of returning the result tiles of the first list that satisfy the query at the magnification level and the determination that the magnification level of the query is less than or equal to the first threshold, the next lowest magnification level tile is extracted. The process of returning the result tiles of the second list that satisfy the query at the next lowest magnification level, and the process of processing the result tiles of each list, for each of the result tiles, the result tile, the result. Adding to a subset of tiles, recursively searching the subset in response to determining that the total number of result tiles in the subset is greater than or equal to the second threshold, recursive search for each of the subsets. The process comprises storing the search results in the remaining subset, recursively searching the remaining subset, and storing the search results for the remaining subset. Provide a method.

In one embodiment, each magnification level corresponds to one level in the quadtree, and each level in the quadtree comprises a tile representing an image result. In one embodiment, the child tile is associated with the parent tile. In one embodiment, the parent tile is at least one of the downsampling and lowpass spatial filtering of the corresponding at least one child tile. In one embodiment, the process of removing tiles at the next lowest magnification level comprises producing children at the next lowest level in the quadtree. In one embodiment, the query comprises at least one of a minimum threshold for the number of results and a maximum threshold for the number of results. In one embodiment, the threshold for the number of results is based on system resources. In one embodiment, the query includes an image and the magnification level of the query is the magnification level of the image. In one embodiment, the result tile is based on at least one of the magnification of the result tile, the image of the query, the filename of the index associated with the result tile, the size of the result, and the type of index. Included in the returned list. In one embodiment, the query includes a time limit for performing a search within a time limit. In one embodiment, the query comprises a quality level threshold. In one embodiment, the predetermined first threshold is defined so that the method ends in response to a determination that the number of search results falls below a certain number. In one embodiment, the predetermined first threshold is defined to correspond to a numerical value at the magnification level. In one embodiment, the given first threshold is defined so that depth-based searches are not used. In one embodiment, a magnification level has a higher resolution than a lower magnification level. In one embodiment, one magnification level is that the resolution of each dimension is twice the next lowest magnification level. In one embodiment, the query is updated after the process of returning the result tile of the first list. In one embodiment, the method further excludes the result from at least one of the first list result tile and the second list result tile before the process of processing each list result tile. Including the process of doing. In one embodiment, the process of processing the result tiles for each list further clears the subset after storing the results of each recursive search, and the result of the recursive search for the remaining subset. Has to clear the remaining subset after saving. In one embodiment, the recursive search is a depth-first search. In one embodiment, the stored results are available before the search is finished.

In one embodiment, methods and systems for performing a recursive search for tile sets based on query tiles start at the next level for each tile in the tile set until the result set is populated. In the process of taking out a set of tiles and adding the next level tile set to the result set, and in the case of determining that the magnification level is a predetermined target level, within the result set. For each tile in the result set, the number of tiles in the subset is the third in the process of assessing the quality of the match and determining that the magnification level is below the target level. Reciting the subset in response to the determination that it is greater than or equal to the threshold and adding that tile to the subset, and in response to the determination that the number of tiles in the subset is less than the third threshold. To search by Includes the process of adding results to the provisional result set, clearing the subset, and returning the provisional result set.

In one embodiment, the query tile is included as the first child tile of the first tile in the tile set. In one embodiment, the process of evaluating the quality of a match is to determine whether the match value of the first tile in the result set compared to the query tile is less than a predetermined value. Have. In one embodiment, the predetermined value is 50%. In one embodiment, the quality of the match is based on the differences between the vectors. In one embodiment, the difference between the vectors is based on the distance between the vectors of the respective tile. In one embodiment, the difference between the vectors is based on the mean square error between the vectors of each tile. In one embodiment, each pixel in a tile has at least one number representing at least one of the color and brightness of each pixel, and each tile has all the pixels in that tile. Has a vector of at least one numerical value with respect to. In one embodiment, the query comprises said predetermined values for assessing the quality of a match. In one embodiment, the tentative result set returned is sorted based on the degree of matching with the query. In one embodiment, the interim result set is sorted based on the degree of matching with the corresponding parent tile.

In one embodiment, the processing execution plan executed by the computer is with at least one selectable probe feature designation, including at least the spatial position and extent of the image features, and at least one target designation, including a set of images. To generate a correlation sample of at least one targeting, including at least one image of the microscope slide, and the at least one selectable probe feature and the at least one targeting. The correlation sample comprises a traversal plan, including a comparison order and a comparison operator of, the similarity method, a similarity measure, said at least one selectable probe feature designation, said at least one target designation, and said. Includes said spread of image features. In one embodiment, the traversal plan includes at least a method of ordering samples and applying a similarity measure to establish a correlation with said at least one selectable probe feature designation, and the data comprising said correlation. Is kept in persistent computer memory available for the traversal plan so that the traversal plan adapts the processing execution plan to evaluate the correlation. In one embodiment, the traversal plan samples the sample with statistically uniform sampling with a uniform lattice, quadtree decomposition of the slide, embedded zero tree of the slide, exhaustive sampling, sparse. ) Sampling, and scale / proximity biased sampling shall be based on evaluation in the order defined by at least one of them.

In one embodiment, the transitional bias is applied adaptively so that the correlation with each data can be predicted by using at least one correlation with the previously correlated data. In one embodiment, online machine learning is used to bias sampling and said traversal plans. In one embodiment, relevance feedback from the user is used to bias sampling during the execution of the traversal plan. In one embodiment, the traversal plan defines result parameters that can be used to determine which samples are returned as part of the result set, the parameters including magnification scale and spatial extent, said process execution plan. Defines the order in which the samples are evaluated, which determines the rate at which the samples in the result set are returned.

In one embodiment, scale-based dependent trees are defined based on the separation of their respective evaluation states, and the scale-based dependent trees (trees) are distributed to separate processing elements for parallel evaluation. In one embodiment, a defined section of the data is given independently of the other data sections and an intermediate set of data is generated for partial results. In one embodiment, the divided data and the processing designation are stored in the same storage device. In one embodiment, the split is based on the number of resulting samples returned for each sample evaluation, the split size being at least one of increase and decrease. In one embodiment, the conversion process is applied to at least one image processing conversion to the result set, and the output of the conversion process comprises a converted sample stored in a permanent storage device.

In one embodiment, the probe and target samples are from the available converted and secondary samples so that the sample returned by running forms a traversal plan that is a secondary result sample. Be selected. In one embodiment, the secondary result set sample is used to adaptively bias the primary traversal plan, and a strong correlation based on the secondary result set is the primary traversal. Indicates that the corresponding sample in the plan should be evaluated preferentially. In one embodiment, the secondary result sample is adaptively biased to further transform the secondary result sample to produce a transformed tertiary sample, said primary result set. The adaptive biasing of the secondary result set to the secondary result set also extends to the bias application of the secondary result set by the tertiary result set. In one embodiment, adaptive bias application between the upstream plan and the downstream plan is used in a chain. In one embodiment, graph topography is utilized for the adaptive bias application.

In one embodiment, a method of continuously processing an image data repository performed by a computer includes a process of receiving a query specification including a request for data and a process of receiving a system designation of the computer performing the method. , The process of comparing the query designation with the system designation to determine the domain designation, the process of initiating a query for the repository based on the domain designation, and the results of the query, including image data. The process of receiving, the process of displaying the interactive and iterative search of the result image data on the graphical user interface, the process of receiving the input of the result image data via the graphical user interface, and the process of receiving the query. It includes a process of updating based on the input, and a process of displaying the updated graphical user interface based on the updated result image data. In one embodiment, the repository of image data comprises digital microscopy data. In one embodiment, the repository of image data includes tissue image data of a certain scale so that the approximation of the data on the coarse scale side does not correlate with the data on the fine scale side. In one embodiment, the continuous processing of the image data progressively produces results so that the results are available before the processing is completely finished. In one embodiment, the query designation implicitly defines indexing and transformation of data.

In one embodiment, a computer-executed method of transforming image data in a data repository based on a query is a process of receiving a query for the data in the data repository, wherein the query is at least one. Each process, including the probe tile and a slide of at least one group in which the query is executed, and the data repository, as long as the overlap between the query and the probe tile is spatially relevant. A process of recursively searching through the magnification level, a process of refining (narrowing down) the query based on the results of the recursive search, and a traversal plan for the target slide based on the recursive search. The transformation comprises the process of generating and transforming the data based on the data returned by the query, the transformation adjusting at least one of the individual pel positions of the data and the color depth of the data. Have to do.

In one embodiment, the overlaps are considered to be spatially relevant in response to the determination that there is an overlap of at least 256 pixels. In one embodiment, the query comprises a search predicate and a query target. In one embodiment, the probe feature designation comprises a search predicate designated as a region of interest, said region of interest comprising a position on the image having a specified extent. In one embodiment, the probe feature designation comprises at least one tile having a set of images targeted by the probe feature designation in generating a search match. In one embodiment, the traversal plan comprises an order in which the targets are compared with at least one probe feature designation. In one embodiment, each of the probe feature designation and the target is at least one of a tile, a single image, and a sub-image of a microscope slide image at a certain magnification level.

It is a flow diagram which shows the method of the interactive data search in one exemplary embodiment. It is a flow diagram which shows the search life cycle in one exemplary embodiment. It is a block diagram which shows the query unit in one exemplary embodiment. It is a block diagram which shows the analysis unit in one exemplary embodiment. It is a figure which shows the architecture in the intermediate stage of a query unit in one exemplary embodiment. It is a block diagram which shows the component of the repository in one exemplary Embodiment. It is a figure which shows an exemplary embodiment. It is a figure which shows an exemplary embodiment. It is a figure which shows an exemplary embodiment. It is a figure which shows an exemplary embodiment. It is a figure which shows an example of the slide layout in one Embodiment of this invention. It is a figure which shows an example of the feature comparison of a slide or a tile in an exemplary embodiment. It is a figure which shows an example of the spatial decomposition of a tile in one example embodiment. It is a figure which shows an example of the scale decomposition of the tile in one example embodiment.

It provides methods, system and computer readable (stored in storage media) instructions for continuously processing data (eg, image data) to address problems arising from semantic gaps. In one embodiment, the method is invoked by a query (which represents a user's intent), a function of the system, and a guide of the user by the system. The method provides an interactive and iterative search for data via at least one of querying, analyzing and processing the data. For example, the present invention provides a search for image data that focuses on the unique requirements of tissue image data and other unique requirements of biometric data. Various uses can be considered as the uses of the present invention. For example, the present invention can be applied to any image in any industry, including photography, satellite imagery, and the like.

In one embodiment, the search method and system provide the user with immediate feedback on the query scope and query results. This facilitates immediate refinement of the query. The method can further specify the analysis or processing to be performed on the image data results returned by the query.

In one embodiment, the query specification implicitly defines the derived index and data transformation of the data. In one embodiment, the definition produces a result for a query. These results are in response to the results from that definition. In one embodiment, the results are pre-calculated and / or on-demand calculated and / or calculated during a past search. In one embodiment, the results are incrementally returned based on at least one of user experience requirements, user query specification / refinement, and system functionality. In one embodiment, a plurality of query processes, analysis processes, and processing processes are chained by iterative processing of each of these processes. The combination of these processes becomes the processing pipeline.

In one embodiment, the components of the system and method provide the means by which the system continuously solves the processing pipeline. In one embodiment, the result of the process is incrementally generated and given to the user and / or later in the pipeline. This configuration is advantageous because it is not necessary to completely process all the data in one pipeline process. For example, the results are provided each time they are found by at least one processor. If a given result is selected as more relevant, the query will be updated with this information, and subsequent searches and results by at least one processor will include the updated query. It is said that. Then, the at least one processor continues the search that has been started. In one alternative embodiment, when the query is modified, a new search is initiated.

In one embodiment, the systems and methods include archiving, storing, transferring and analyzing operations and are prioritized in this order. Archiving operations include retention and duplication of data (eg, tissue image data). This makes it possible to ensure that the data can be stored for a long period of time without being frequently moved. Performing processing on data that remains archived does not move the data to the side local to the arithmetic processing of the data, but rather moves the arithmetic processing of the data to the side local to the arithmetic processing of the data. Accompanied by moving. The storage operation provides access to the data in a multi-scale decimation representation. The storage operation organizes the data to reduce access latency and manage the storage and loading of derived data. The transfer operation limits the need to transmit data, i.e. derived data. For example, the transfer operation delays the transfer of data to downstream processing and performs analysis and conversion of the data locally on the data and returns results when the derived data is small.

FIG. 1 is a flow chart showing an interactive data search method and system 100 in an exemplary embodiment. The method and system 100 include a sequence of processes for searching for information in the repository. In step 102, the user designation is defined based on the user's data extraction requirements. In step 104, the user designation defined in step 102 is crossed with the domain-specific information in the repository. This will generate the relevant domain designation in step 106. The query for the repository is started using the combination of the user designation (step 102) and the domain designation (step 106). As a result, the query result is generated in step 108. The generated results are fed to the analytical process. In step 110, this analysis process produces an analysis of the results.

FIG. 6 is a block diagram showing components 600 of the repository according to an exemplary embodiment. In this example, the components of the repository include the basic data 602 of the repository. The basic data 602 is a quadtree decomposition of slide image data, specifically, each parent layer is subsampled once and each layer is divided into tiles corresponding to the tiles of the tree components (eg, each layer). It is a quadtree decomposition that has been thinned out. The tile indexing 604 is defined based on at least one feature extraction 608. The feature extraction 608 is separated into the states of each tile. Correlation index 610 is defined based on index similarity and indicates the correlation between two or more tiles based on at least one index 604. The basic data 602 is a conversion 606 of the imported data.

FIG. 2 is a flow chart showing the repository search life cycle 200 in one exemplary embodiment. In one embodiment, the search is a combination of queries and analyzes that are sequentially executed by at least one query unit 202 and at least one analysis unit 216. For example, at least one query unit 202 includes a query target. For this query target, an index 208 and / or a correlation 210 is identified based on, for example, feature extraction. Index and correlation information is stored in the repository 212. The result 206 obtained via at least one query unit 202 is sent to analysis unit 216 and / or refinement unit 204. After the analysis unit 216, the result is sent to the conversion unit 214. After the conversion unit 214, the result is sent to the repository 212. Various information stored in the repository 212 is sent to at least one query unit 202. A refined version of result 206, 204, is also sent to the query unit to update the search.

In one embodiment, at least one query unit 202 includes a search predicate and a query target. As used herein, the term "probe (search)" or "probe feature designation (search feature designation)" refers to the search predicate designated as a region of interest. A region of interest is, for example, a point on an image that has a specified extent. As used herein, "targeting" refers to a set of target images that the probe targets in generating search matches and / or images that make up the tiles on at least one microscope slide (eg, digital images). Refers to the set of. In one embodiment, when at least one target image or target tile is selected, that selection becomes the probe. As used herein, the term "traversal plan" refers to an assembly in an order in which targets from a target designation are compared to at least one probe. In one embodiment, each of the individual comparisons is a comparison of one probe and one target. In one embodiment, the comparison is a comparison of one or more probes with one or more targets. In one embodiment, the probe and target are at least one of a tile, a single image, and a sub-image of a microscope slide image at a particular scale or magnification. In one embodiment, the traversal plan is breadth-first traversal for multiscale quadtree decomposition of microscopic slide images. In one embodiment, the traversal plan is a depth-first traversal for multiscale quadtree decomposition of microscopic slide images. In one embodiment, the traversal plan is a depth traversal followed by a breadth traversal. In one embodiment, the traversal plan is traversal in the width direction followed by traversal in the depth direction.

In one embodiment, the user (eg, pathologist, administrator, processor, etc.) selects at least one probe tile to be used to query the tissue image repository. The user further selects a group of slides on which the query is executed. When the query is executed, features are extracted from at least one probe tile image based on the query specification. For example, color histogram feature extraction is used. In one embodiment, when a feature is extracted, it is permanently maintained in long-term storage to prevent recalculation of the feature. A collection of at least one feature vector permanently maintained on an disk is defined as an index. A feature vector is extracted from the lower magnification scale of the slide that includes the probe tile. In one embodiment, the process recursively scales the lower magnification until the last level (eg, the highest level) where the overlap with the probe tile is spatially relevant is reached. go back. In one embodiment, this spatial relevance is an overlap of 256 pixels. Extracting features from the highest level probe overlap tiles for individual probe tiles, and extracting features from intermediate magnification level probe overlap tiles for individual probe tiles, but for that individual probe tile Defined as a scale probe tile set for. In one embodiment, a collection of all the tile sets for all the probe tiles (collectively, the total probe tile set) is used to generate a traversal plan for the target slide. In one embodiment, the components of this total probe tile set are combined based on their magnification level. A collection of such components (eg, features extracted from a tileset) is used to order the set of extracted features of a candidate target tile in generating a traversal plan. In one embodiment, features from the target tile are extracted at the corresponding magnification level and compared to the probe set. The plurality of target candidates are then ordered based on the feature vector similarity measure. In one embodiment, the comparison operator is the L2 norm of two vectors. In one embodiment, the traversal plan is this ordered list, traversed in order of similarity, and the tile at the next highest magnification is the feature space defined as the corresponding probe set tile (eg, this example). In terms of color histogram), they are compared. The results are also ordered, and then recursive processing continues to a stronger magnification level. In one embodiment, the traversal plan is designated as breadth-first or depth-first, with higher magnification levels recursively processed before all evaluations of the current magnification level are complete. Determines whether the higher magnification level is recursively processed after all evaluations of the current magnification level have been completed. For example, a depth-first one has the advantage of being able to provide results to the user with less latency because less evaluation is performed.

In one embodiment, the result returned from the execution of the query using the traversal plan is displayed in the user interface. In one embodiment, the user adds or removes probe tiles, adds or removes target slides, and inputs relevance feedback on the returned results at any time the results are displayed in the user interface. You can choose to change the query parameters. In one embodiment, when probe tiles or slides are added / removed, a simple process set in the existing set of probe tile features is applied, thereby modifying the traversal plan. In one embodiment, the relevance feedback not only alters the traversal order, but also serves as a bias factor for the similarity measure. The relevance feedback is designated in the user interface as a positive (+) corresponding to positive feedback or a negative (-) corresponding to negative feedback.

In one embodiment, the user can specify at least one additional query that targets the result of the first query during execution of the query using the traversal plan. Specifically, the later query affects the results of the previous query and searches the result set. In one embodiment, the primary query is based on the extraction of color histogram features, and the secondary query is based on a more complex feature-based query (eg, based on texture characterization). In one embodiment, the secondary query is based on texture orientation (eg, histogram feature extraction by sparse Gabor).

In one embodiment, the user specifies an analytic transformation to be performed on the results of the query. Specifically, the conversion process, when the query result is generated, converts the resulting tile to the converted version. In one embodiment, the transformation performs an image processing morphology process that expands or contracts the image features in the result tile to show spatial support. In one embodiment, the conversion process is a deconvolution process that separates the colors corresponding to the stains used in preparing the imaged tissue. In one embodiment, a tissue quantification process is performed that identifies tissues such as cell nuclei, stroma, and glands and determines the location of the tissues. In one embodiment, the positioning and identification of such structures is used to transform the resulting tile and amplify the targeted cellular structure.

In one embodiment, the query results are passed to a defined analytical process that produces at least one converted result for each returned result each time it is incrementally generated. In one embodiment, the process retains the transformed result for processing and querying operations. By holding in this way, those tiles can be used in the same way as the original tiles. For example, at least one converted tile can be specified as a query tile, and each time a converted result is generated, a query operation is performed over the converted result. The converted results are generated from the original query behavior on the original tile. In one embodiment, the relevance feedback action and / or the query tile addition action can be performed during query execution.

In one embodiment, the combination of the query and the analytical transformation is chained together so that the output of the query process is the input of the transformation process and the output of this transformation process is the input of another query process. In one embodiment, this chain of query / analysis processing is virtually unlimited. In such a processing chain, for example, the process is initiated by the user selecting some query tiles from the base layer of the image. The query is then executed when the user then selects the slide to which the query is targeted, then the index used for that query (eg, the index based on the color histogram) is selected, and the user chooses to execute. In each of those tiles, the query begins to generate results, the user specifies to perform an analysis on the results, and the user specifies that the analysis performs color decoding on the result tiles. Is converted to another tile (eg, H & E (hematoxylin & eosin) dyed tile), and the conversion result is generated and given to the user. Each time this query process produces more results, the results are automatically converted and given to the user. The user then selects one of the converted tiles (eg, the one corresponding to hematoxylin staining) as the query tile. The index for this query is chosen based on the texture feature extraction. When this query is executed, the result is determined from multiple converted hematoxylin tiles. As each step in the above chain produces more results, this process returns more results.

In one embodiment, embodiments of the process described herein are performed when new slides are added to the image repository. In one embodiment of this case, the similarity scale of the final result is thresholded, and the result exceeding the alert threshold is sent to the alert generation system. In one embodiment, the processing pipeline is triggered by adding a slide to the repository, and an alert notifies the user for evaluation of the new slide and the result set tile associated with that pipeline. This is an automated screening process for scanned slides that leverages existing processing pipelines to automatically process slides added to the system and generate notifications or alerts for further processing. Such automated processing can be applied to screen slides for a variety of purposes, including abnormal tissue detection and quality assurance.

In one embodiment, the query tile is specified from the source slide (the source slide). Scanned specimen slides are stored in an arrangement of tiles arranged on multiple different scales. The highest scale is the original magnification at which the slide was imaged. Typically, this magnification is an optical magnification of 40x (hereinafter, "x" may be described as "x (multiplication operator)"). High-scale tiles are subsampled to low scale, and each low scale is taken to be 1/2 the magnification of the previous scale. For example, if the highest scale is 40x, the next highest scale is 20x, and then 10x, 5x, 2.5x, 1.25x, 0.62x, 0.31x. For example, on the lowest scale, a typical tissue sample occupies 4-8 tiles. Each tile has 256x256 pels. In one embodiment, a target slide is specified to perform a search for a match with the query tile. For example, the order in which the target tiles are compared to the query tiles is an exhaustive traversal of the target tiles at the same magnification as the query tiles, using a comparison operator based on the L2 norm of the pels between the tiles to be compared. It is an order. In one embodiment, a multi-scale search / comparison is performed that uses at least one available low magnification scale to generate a match hypothesis and then increases the magnification until the target magnification is reached to prove the match hypothesis. ..

The magnification is halved for each continuous scale, and the number of pels for each dimension is also halved. In this way, the collection of tiles on each scale is a subsampling of the previous higher magnification scale. One tile on a scale corresponds to four tiles at the next highest magnification. This correspondence matches the quadtree decomposition of the original full-scale slide image at the highest magnification. The present invention traverses this quadtree, where each tile is a node (node) of the tree and each spatial correspondence is an edge (branch) of the tree.

In one embodiment, such a quadtree, traversal for multi-scale search, has a spatially corresponding tile match at a low magnification and the presence of match candidates at a high magnification up to the query tile magnification. It is used as a suggestion. In one embodiment, the underlying traversal determines the priority order of further comparisons with respect to the subtrees of the tiles at the lowest magnification by comparing the corresponding tiles at the lowest magnification with the query tiles. In one embodiment, the tiles with the lowest magnification levels are ordered on a similarity scale. In one embodiment, at least one tile with the lowest similarity is excluded based on its similarity below a given threshold. In one embodiment, this threshold is the 0.70 correlation of the feature vector derived from the tile. In one embodiment, the feature vector for each tile is a histogram of 32 bins based on the sum of the spectral components of each tile. For each of the retained tiles, the corresponding four tiles at the next highest magnification are added to the new collection of tiles. This collection of tiles is evaluated based on the same process as the previous level, and this recursive process continues until the base magnification is reached. In one embodiment, the correlation threshold starts at a smaller value of 0.50 and gradually increases with each magnification level up to a maximum of 0.80. The embodiment described here is breadth-first traversal for quadtrees, which is the next highest after a comparison of the corresponding query tile at low magnification with the corresponding search tile at its current level. Move on to magnification. This is equivalent to recursively processing the next level based on all matches at the current level in a single batch. In one embodiment, such recursive processing is performed on a plurality of batches obtained by dividing this single batch evenly. Each of these small batches is also recursively processed to a higher magnification level. In an embodiment where the batch size is one tile at the current level, the traversal for the quadtrees described above is a depth-first traversal for the quadtrees. When the batch size for each level is variable between a single batch (breadth-first) and one tile per batch (depth-first), it is called adaptive traversal.

In one embodiment, the adaptive traversal is set to breadth-first at the start of the search. As the search progresses, the calculation cost is calculated as the number of similarity operations performed. In terms of computational cost, for example, the number of search results returned for each comparison determines the incremental search result latency. As an example, if the number of similarity comparisons is 200 and the number of results returned when the correlation similarity scale is 0.90, the comparison: result ratio is 200:40. That is, there are five result latencies for each result. In such a situation, breadth-first division, which divides all tiles at the current level into a single batch process, is determined to be efficient. If the resulting latency increases above a given threshold (eg, 50), this signals the adaptive traversal mechanism to increase the number of batches at each level (eg, to two). .. If the result latency still exceeds the threshold, the number of batches for each level is increased to three. The threshold is set to any numerical value desired or required by the administrator or system. In one embodiment, the adaptive mechanism sacrifices processing efficiency at each level to target results with high similarity before moving on to results with low similarity (depth-first processing). .. At some point, the number of batches may be equal to the number of tiles at the current level, which corresponds to a strict depth-first traversal for quadtrees.

In one embodiment, the adaptive traversal incrementally divides the batch process to increase the number of batches at each level in order to reduce the resulting latency. For example, if the process reaches the depth-first batch split limit but the result latency increases, the process gradually reduces the number of splits for search in order to reduce the result latency. In one embodiment, a computational budget is specified to determine the number of subtrees to be evaluated.

In one embodiment, a sampling function for accessing data (for example, tissue image data) is provided. This sampling function defines the order in which data is accessed. This feature builds an access plan based on the constraints expected from the data itself and the user's interactive behavior. These constraints on the sampling function constitute the query context according to the present invention.

In one embodiment, sampling consists of user-specified and system-specified access plans. The user designation includes a range of data to be searched in addition to any predicate designation. The system designation includes existing data and remaining results from past processing. The sampling function incrementally returns a set of partial results. The results returned in this way are used to modify the user and system specifications within the current query context. Sophistication behavior interrupts the process so that the query is directed to a more relevant result set (eg, a sample).

FIG. 3 is a block diagram showing a query unit 300 in an exemplary embodiment. The query unit 300 includes a domain (system) designation 302 and a user designation 310 for queries. The domain designation 302 is combined with the user designation 310 to form a predicate and / or target 304, and the predicate and / or target 304 produces result 306. Refinement and / or extension (308) of the query based on the results modifies the user-specified 310 of the query.

FIG. 4 is a block diagram showing the analysis unit 400 in one exemplary embodiment. The analysis unit 400 generates conversion results 406 and 408 based on the data returned from the query and / or user-specified 410 in the repository. The converted data is in the form of an image having the same dimensions and the same resolution as the data from which the data is derived, and the individual pel positions and color depths are changed based on the conversion. Domain designation 402 examines the results and confirms and / or transforms the results.

FIG. 5 is a diagram showing the architecture of the query unit 500 in the intermediate stage in one exemplary embodiment. An intermediate result (“intermediate”) is the result of a defined process performed to produce a result. The intermediate depends on the combination of the basic state of the repository and the defined treatment.

In one embodiment, the region of interest (“ROI”) 502 defines at least one adjacent pel in the repository that constitutes one tile 512 in the repository. ROI 502 may be a complex polygon 522. Such a polygon 522 is defined by at least one tile, and the inside of the polygon 522 is interpolated to discrete positions on one or more tiles. The tile data is the image data thinned out in scale and space, and generates the basic unit of processing of the tile. Vector 514 is a feature vector 504,516 extracted from the tile and is used to generate the index on which the query acts. The correlation intermediate 506 describes the correlation between the two tiles as the similarity between the extracted feature vectors of the two tiles (eg, one of the two tiles is a query predicate and the two tiles It is retained by the similarity generated when the other of them is the query result). The correlation itself is the feature index to be searched. The tensor 526 is constructed to convey the correlation to other feature vectors. Layer 508 is any post-transformation (518) and / or post-filtering and / or post-visualization information derived from the data. Quantification 524 is a quantitative escalation process involving, for example, aggregate tile processing (eg achieved by scale-based constraints). Scale 528 is any of scale-based constraints, thresholds, correlations, indexes and other information.

In one embodiment, the structure and content of the data defines the order of priority of processing, which determines the order in which the data is given to the user. This allows the user to understand both the structure and content of the data. The user is guided to search for data by this priority order. This reduces the need for prior knowledge of the data.

In one embodiment, the process is tuned for the returned data and extends the sampling range of the retrieved data based on the return of a sparse set of results. Similarly, in one embodiment, the sampling range is limited when a large amount of data is returned. Such restrictions allow for extensive sampling of data while preventing large amounts of data from being returned for a small localized subset of all data.

In one embodiment, spatial continuity yields the presumption that spatially adjacent data is generally more relevant than distant data. Similarly, when the correlation between the nearby data and the distant data is large, the correlation between the nearby data and the adjacent data is also large for the same reason. In one embodiment, these relationships are used as the basis for predicting search constraints.

In one embodiment, usage statistics extend the processing range of generated and accessed data to a wider range than other data. If data is generated and accessed infrequently, intermediate data restriction and flush operations are performed. The frequency of access affects the ranking of data. IAPE (Ranking Data for User Search) is adjusted for quality assurance (“QA”) by bias compared to ranking based on system processing (eg, bias based on slide screening, etc.) Will be done.

In one embodiment, the system and method provide means for extending and / or limiting the result when it is returned to the user. This online tuning feature provides the user with a means to interact with the query results. This feature is made available to the user at any point in the query execution process (even after the query is finished). In such a case, the query will be re-executed with the new constraints. In one embodiment, the query and the incremental results of the query are analyzed to determine if a given limit has been reached that makes processing continuation of the query inefficient. If it is determined that the limit has been reached, the query may be stopped and the user may be given the opportunity to make changes to the query.

In one embodiment, data partitioning and traversal are performed to optimally maintain memory and computational connectivity. In one embodiment, the data is regularly divided into spatial regions that do not overlap each other (eg, blocks, tiles, etc.). In one embodiment, the data is subsampled and divided. In one embodiment, the divided data tiles are arranged based on scale and spatial proximity. The localization of each individual tile serves as the basic unit of operation. This basic unit is processed and the result is given to the user.

In one embodiment, aggregate tile processing is primarily achieved by scale-based constraints that examine the suitability of data processing order at high scale using analysis for low scale. For example, a scale-based pyramidal structure of tissue image data is represented as a quadtree decomposition of the image data. Parent-node similarity is used to test the suitability of this expression. Increasing access to side road information yields a large number of step-by-step results. This means that you only have to run part of the pipeline.

In one embodiment, the processing of the algorithm depends on the amount of results returned. The traversal policy determines how the traversal strategy will change.

In one embodiment, the organization of image data storage and the organization of derived data thereof can be expressed by quadtree decomposition, and the particle size of each processing gradual increase is based on the processing of the subtree of the quadtree. The operations required to return a set of results are limited using subtree processing cost estimates. Separation of processing into subtrees makes it easier to apply distributed parallel processing to scale traversal operations.

In one embodiment, traversal for a quadtree is defined to process a subset at each level, and the number of subsets at each level is the degree to which the traversal is breadth-first or depth-first. To determine. Breadth-first bias allows a large amount of data to be sampled and uses a large amount of computational resources to return a set of results. Such a gradual increase method is advantageous when the matching result is sparse and the scale association is weak. In one embodiment, breadth-first traversal is generally exhaustive and does not make many assumptions about the distribution of matching samples. This suggests that the predicate search hypothesis is weak. In one embodiment, a depth-first bias samples a smaller amount of data, results in shorter computation times, and returns results with less gradual increments. This is advantageous when the results are dense and the scale association is strong.

In one embodiment, the search for data produces a number of partial solutions for subsequent queries. These partial decompositions represent an opportunity to provide results faster than the results calculated from a small amount of intermediate data. There is a quality bias where many of these partial results are generated by the activities of other users. It also returns a subset of the results with this bias.

In one embodiment, this quality bias is not only available to improve the efficiency of returning results, but is also recursively referenced (eg, counted) on a data-by-data basis as a comprehensive ranking of data usefulness. ).

In one embodiment, the problem of tissue image data is addressed by system compatibility, which allows user-specified sophistication in addition to downstream processing in the pipeline. This refinement of specification is used to modify the current query processing to change the results produced by the query. Downstream processing, when a query result is generated, acts on the query result to perform additional transformations on the data. After that, additional query processing is performed. Query processing responsiveness can provide users with the flexibility to explore data through query changes and further processing.

Basic image data is structured and organized for incremental processing across spatial and spectral scales. The data is spatially and spectrally normalized by the calibration process upon import. The correlation of the data is determined by the similarity of the feature indexes and is maintained at the correlation index. Data access / processing is estimated, and such access processing is performed based on the predicted cost and the actual cost. A preliminary operation that estimates the result of the entire operation is performed to provide an estimate of the operation cost and to incrementally calculate the final result. The results are rolled up and aggregated for future operations, and intermediate products are retained for update operations that allow online operations on the algorithm.

In one embodiment, the kernel facilitates gradual and separate operations by utilizing pyramidal / hierarchical / quadtree data structures (eg, multiscale image pyramidal structures). do. In one embodiment not limited to the present invention, the tile is a square image with a spatial extent of 256 in each dimension. These tiles are generated from the original image by performing consecutive 2x subsamplings of the original image until the dimensions of the recursively subsampled image are less than 256. The file system directory containing these tiles is also divided into 256 groups, or 16x16 tile layouts. This file system organization provides the optimal placement configuration for the localization of the storage system to utilize the cache storage mechanism. Furthermore, such a separate configuration makes it possible to anticipate a distributed file system that can perform processing on sub-areas without having to share context information between different computing environments. ..

In one embodiment, a histogram bin-based feature vector is used in tile-to-tile similarity comparisons. Such a feature vector is an approximation of the tile content and represents the approximation used for indexing. For tiles that are similar to each other in such a spectral feature space, the processing for these tiles is used to estimate the result of processing involving feature extraction processing, which is more computationally intensive.

In one embodiment, the scale aspect of the technique according to the invention emphasizes that the spectral feature vector, which is an approximation of the spectral components of the tile, is a ranked approximation. For example, when the feature vector is the size of a tile, the position of each feature vector corresponds directly to each pixel. In one embodiment, if there are fewer bins than pixels, it inevitably means that the scale is downgraded from the original data. If there is no correspondence between the bin of the histogram and the position of the pixel, it means spatial invariance.

In one embodiment, the kernel runs online, performs coarse-grained operations and aggregates the results. The calculation cost of executing those operations is incorporated as a factor in the processing scheduling operation. Cost estimates are performed for the access plan subtrees and they are aggregated. These estimates allow for adjustments to the operations allowed for each subtree.

In one embodiment, the kernel is configured to act incrementally on the pipeline of processing elements. These elements may be such that the query element is followed by the analysis element. The query element applies at least one predicate pattern to a set of candidate patterns and returns a result based on a pattern matching condition. Analysis is performed on the result set pattern and somehow transforms the result set pattern for at least one of visualization, further analysis and query behavior.

In one embodiment, the query itself produces intermediate products based on the index generation, and these indexes are used in the pattern matching process. In one embodiment, the analytical process produces an intermediate product in the form of filtered or transformed input image data or a quantitative scale.

In one embodiment, another intermediate product may also include an intermediate product from an approximation function that produces an approximated result. Also, as another intermediate product, an intermediate product from an online operation is retained for the purpose of increasing the iterative processing speed. Intermediate products from such online operations are also considered intermediate products in the pipeline.

In one embodiment, the retention and utilization of these intermediate products allows the kernel to take an alternative approach of producing results without the computation required to repeat these processes.

In one embodiment, data such as tissue image data has different structures on different scales. These structures are not necessarily structurally linked across different scales. That is, the structural pattern is not always repeated. The relationship between these patterns may be modeled as a generation function that allows the macroscale model to generate at least one microscale model. These models can be used to impose additional constraints on queries. A kernel-specific aspect of these models is that the kernel finds macro / micromodels such as those described above regardless of specific data, and the kernel uses the macro / micromodel to collaborate over different scales. To provide similarity.

In terms of data management in one embodiment, the usefulness of the system depends on the nature and organization of the data. In tissue image data applications, retention of the original image data takes precedence over retention of derived data. The data management system configuration reflects the priority order of the archive and defines the storage operation, the transfer operation, and the analysis operation with reference to this priority order.

In one embodiment, the size of the data imposes a substantial limit on the data replication operation. Considering this size together with the long-term retention policy associated with the data, it is possible to meet the requirements by building the database around the archived data. The choice of archive format and data layout affects the functionality of all downstream processing.

In one embodiment, the storage of data is based on the ability to manage multiple different layers of data derived from the data. This data retention and flushing operation is performed to satisfy storage and computing requirements based on the ability to regenerate such data on demand.

In one embodiment, the processing associated with the transfer and distribution of data is accomplished by the physical grouping of the data and the ability to make out-of-range references resolved by a given addressing technique. By imposing restrictions on dependencies, the system has an operational advantage when using processing in a distributed environment.

In one embodiment, the system has actions that are automatically performed based on both routine processing and user interaction.

7A and 7B are simplified flow diagrams showing how to initiate a query to retrieve data in one exemplary embodiment. As shown in FIG. 7A, after the search is started (block 705), the current zoom level (or "magnification", used synonymously) of the image for image query is greater than the predetermined threshold Th1 (block). If 710), search results for the current level of query are returned (block 715).

In one embodiment, the threshold is set to limit unproductive search operations, eg, to end the search if the number of search matches generated is not large. As another example, the threshold is set to limit unproductive search operations before the pixels of the image no longer contain meaningful image information. As another example, the threshold is set to limit the depth or other depth of the magnification level in the search. For example, if a subset is too small and the number of matches generated from that subset is small, then the size of that subset is increased for a wider search.

In one embodiment, if the search results are pre-calculated or calculated during a past search, the search results are returned without performing a depth-based search.

In one embodiment, the search function is started recursively with different thresholds.

In one embodiment, each zoom level or magnification corresponds to one level in the quadtree. For example, the image is composed of a single tile at magnification level 1. Then, the correspondence between the image and the four children of the image is regarded as the magnification level 2, and the 16 tiles that are children for these four children are regarded as the zoom level 3, and these 16 tiles. ... and so on. Each magnification level has a higher resolution than the previous magnification level, typically doubling the resolution of each dimension. If the tiles reach the resolution limit, the children for those tiles will only correspond if the pixels are interpolated. If such interpolated pixels are also utilized, they are considered to be the maximum zoom level or magnification. The decomposition into quadtrees results in downsampling, lowpass spatial filtering of the corresponding four child tiles, so that the child tiles are associated with their parent tile. For example, if a given color is found in the parent tile, it is likely that the same color will be found in the child tile, or at least the parent color can be derived from the child tile side color by downsampling. be.

In one embodiment, a list of result tiles satisfying the current search condition is queried from memory in order to retrieve the search results. In one embodiment, such a list of tiles is the file name and / or result size and / or of the associated data (eg, tile level and / or query and / or index associated with each tile. Extracted based on index type).

In one embodiment, the current search criteria or query limits search results based on priority and system resources. For example, the query includes limits on the operations available for the search, limits on the time available to complete the search, limits on the amount of memory available, thresholds for the quality or quantity of search results, and so on. ..

In one embodiment, if the current magnification level is not greater than the predetermined threshold Th1 (block 710), the next level of tile is taken out (block 720). This will take out or generate a child of the next level tile, the quadtree, which corresponds to the current level. Then, from those next level tiles, a list of tiles that match the current query result is generated (block 725). In one embodiment, the query is refined as more results are found. For example, the query contains the smallest result size. Then, for example, if a better match is found, a wider result or a given result found earlier is excluded from the result list by narrowing the query that retrieves the result from memory.

In one embodiment, for each tile in the retrieved list, that tile is added to the subset (block 730). If the size of the subset and the number of tiles in the subset are greater than or equal to a predetermined threshold (block 735), a recursive depth-first search is performed on the tiles in the subset (block 740). This reduces the size of the set in which the recursive search was performed. The result of this recursive search is saved (block 745) and the set is cleared (block 750). A recursive search is then performed on the remaining set (block 755), the results are saved (block 760), and the set is cleared (block 765). The saved result can be retrieved from the memory from the time when the result is stored. Thereby, in one embodiment, it is possible to use a set of search results that are continuously updated even during the execution of the search and after the search is completed.

8A and 8B are simplified flow diagrams showing how to perform a recursive search in one exemplary embodiment. As shown in FIG. 8A, a recursive search is initiated for the set of tiles (block 805). To optimize the search, new query tiles are formulated (not shown), for example as the first child tile for the first tile in the input tile set. Then, for each tile in the tile set, the next level tile is retrieved (block 810) and the next level tile is added to the result set (block 815).

In one embodiment, when a result set is created, the quality of matches in the result set is evaluated (block 825) if the current zoom level is the target zoom level (block 820). For example, if the match value of the first tile in the result set is less than 50% compared to the query tile, the result is excessively different from the query tile, so that the search in the depth direction may be performed. No longer continues along this edge (branch) of the quadtree (block 830). On the other hand, if the results are accurate enough, the current result set is returned as the result of the recursive search (block 835). The minimum quality threshold is set as part of the query.

In one embodiment, the quality of the match is evaluated as the difference between the vectors. For example, each pixel in a tile has multiple numbers that represent the color and / or brightness of that pixel. And each tile has such an array or vector of numbers for every pixel in that tile. The two tiles are then compared by calculating the distance or mean square error between the vectors of each tile.

In one embodiment, if the current zoom level is not the target level (block 820), the recursive search continues as shown in FIG. 8B. As shown in FIG. 8B, for each tile in the result set, if the size of the subset is less than a predetermined threshold Th3 (block 840), that tile is added to the subset (block 845). On the other hand, if the size of the subset is greater than or equal to the predetermined threshold Th3 (block 840), a new recursive search is initiated for that subset (block 850). The results of this recursive search are added to the tentative result set (block 855) and a subset thereof is cleared (block 860). This reduces the size of the set in which the recursive search was performed, as described above. Once each tile has been processed in the original result set and these tiles have been added to the subset, a recursive search is performed on the remaining subset (block 865) and this result is the provisional result set. Is added to (block 870) and a subset of it is cleared (block 875). The tentative result set is then returned as the result of this recursive search (block 880).

In one embodiment, the list of one or more tiles at each level is sorted based on the query tiles that match them, or at least one of at least one query tile if not at the target level. Sorted based on the degree of matching with the parent tile.

FIG. 9 is a diagram showing a slide layout according to an embodiment of the present invention. Specifically, living tissue or other specimens are placed on the query slide. From this query slide, a query slide image is prepared. For example, this query slide image is a digital file that is spatially segmented into tiles of square or other shape. From these query slide tiles, at least one tile is identified as at least one query tile (or search tile). Then, the information obtained from the query tile is input to the similarity scale engine. In one embodiment, the similarity measure engine normalizes the query tile to the desired size and memory value.

In one embodiment, the normalization may include any available method. In one embodiment, normalization of at least one tile or slide image comprises obtaining metadata information about micron storage pixel values. For example, if image A is 20 microns per pixel scale and image B is 40 microns per pixel scale, the intermediate levels should equalize the microns per pixel scale of these two slides (eg,). , Both calculated as 20 microns per pixel scale or other levels, etc.). In one embodiment, you may look for the highest resolution capture available, the medium resolution capture available, or the lowest resolution capture available so that different results are obtained depending on the desired image search. .. In one embodiment, color normalization or color correction is performed. For example, the brightness, brightness and color of at least two tiles or images are determined and changed to similar levels for retrieval. This is the same, for example, if the same slide is processed on two different machines (or if two different scanning operations are performed) and the resulting two images are color-normalized or color-corrected. Based on the decision result, it will be possible to correct any other slides from these machines. For example, the brightness, RGB, and other light-based or color-based parameters of two scanned images are compared and a decision is made to change one or both to a particular set of parameter levels, the same source. The same changes or corrections are made in terms of color (color, brightness, brightness, etc.) for subsequent images from.

In one embodiment, the similarity measure engine compares at least one query tile to a target tile located at one or more locations. For example, the target tile is located in one or more physical locations or in one or more databases on the server. For example, at least one target tile of the target tiles is from one or more target slide tiles. The one or more target slide tiles are prepared from a target slide image or a target slide digital image. The target slide image is uploaded from the source and / or generated from at least one target slide. The target slide was prepared using a tissue sample or other sample.

FIG. 10 is a diagram showing an example of tile feature comparison. A query tile or digital image query is entered into the feature extraction engine. This feature extraction engine determines specific parameters and / or features for that query tile. For example, the feature extraction engine uses a given set of features to generate data and / or measurement results for color pixels, sizes, and so on. Some or all of the data and / or measurement results obtained using the feature extraction engine are input to the comparison engine, which is used by the comparison engine for at least one target tile of at least one feature of the query tile. Compare the similarity with the data and / or measurement results of. The data and / or measurement results of the at least one target tile are obtained by using a feature extraction engine or the like.

FIG. 11 is a diagram showing an example of spatial decomposition of tiles in one embodiment of the present invention. A slide image is a digital file or other electronic data or information that can be identified as a slide tile or fragment of the slide image. The slide tiles are then broken down into lower level tile sets.

FIG. 12 is a diagram showing an example of scale decomposition of tiles in one embodiment of the present invention. The slide image is segmented into multiple tiles, one of which is broken down into lower level tiles. One of these lower level tiles is then used to search for other similar tile images, eg, using the various embodiments described herein.

The present invention controls or executes data processing devices such as digital electronic circuits, computer hardware, firmware, software, computer program products, machine-readable storage devices, processors, computers, etc., including the embodiments described herein. It can be realized by the propagation signal. The present invention can be written in any form of programming language, including the embodiments described herein, and can be realized as a component of a stand-alone program or another program. A computer program is deployed and / or stored and / or executed and / or transmitted to and / or one or more of the computers in a single location or in two or more locations. It is possible to send from the computer of.

In the present invention, any process of the method functions by at least one programmable processor, computer, tablet, smartphone, portable smart device, etc. acting on the input data to generate an output. It is done by doing as it does. Storage media include EPROM, EEPROM, flash memory device, chip card, magnetic card, bar code, QR code (registered trademark), DVD-ROM, CD-ROM, internal hard disk, removable disk, magnetic disk, magneto-optical disk, etc. Optical discs and the like are included. The present invention enables interaction with a user by displaying from a computer on a display device, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD) monitor, an LED monitor, a touch screen, or the like.

The present invention processes a large amount of data depending on the implementation. Data is stored in back-end components such as data servers, application servers, and cloud servers, and user interface functional parts such as keyboards, voice input keyboards, graphical user interfaces, and web browsers. In the present invention, the components are interconnected by any form of digital data communication, such as a communication network such as a local area network (LAN) or wide area network (WAN) (eg, the Internet).

It should be interpreted that the description and illustration of the above-described embodiments are merely examples and do not limit the present invention. For example, among the above-described embodiments, different components / components included in different embodiments may be combined or not combined with each other, and may be implemented in various combinations. Various changes, modifications and improvements are possible in view of the teaching contents and the scope of the appended claims, and these changes, modifications and improvements are included in the spirit and scope of the present invention.

Although the present invention has been described with reference to specific examples and specific embodiments, it should be understood that the present invention is not limited to these examples and embodiments. The present invention also includes modifications of the specific examples and embodiments described herein.
The present invention includes the following contents as an embodiment.
[Aspect 1]
A method of searching a database of images, executed by a computer, based on a query.
The process of returning the result tile of the first list that satisfies the query at that magnification level of the query in response to the determination that the magnification level of the query is greater than the first threshold.
The result of a second list in which the next lower magnification level tile is retrieved and the query is satisfied at the next lower magnification level in response to the determination that the magnification level of the query is less than or equal to the first threshold. The process of returning tiles and
In the process of processing the result tiles of each list, for each of the result tiles
Adding the result tile to a subset of the result tile,
Recursively searching the subset in response to the determination that the total number of result tiles in the subset is greater than or equal to the second threshold.
Saving the results of each recursive search for the subset into the remaining subset,
Recursively searching for the remaining subset and saving the results of the search for the remaining subset,
Have a process and
Including methods.
[Aspect 2]
A method according to aspect 1, wherein each magnification level corresponds to one level in the quadtree, and each level in the quadtree comprises a tile representing an image result.
[Aspect 3]
A method in which the child tile has an association with the parent tile in the method according to aspect 2.
[Aspect 4]
The method of aspect 3, wherein the parent tile is at least one of downsampling and lowpass spatial filtering of the corresponding at least one child tile.
[Aspect 5]
The method of aspect 2, wherein the process of removing tiles at the next lowest magnification level is to produce children at the next lowest level in the quadtree.
[Aspect 6]
The method of aspect 1, wherein the query comprises at least one of a minimum threshold for the number of results and a maximum threshold for the number of results.
[Aspect 7]
The method according to aspect 1, wherein the query includes an image and the magnification level of the query is a magnification level of the image.
[Aspect 8]
In the method described in aspect 7, the result tile is at least one of the magnification of the result tile, the image of the query, the filename of the index associated with the result tile, the size of the result, and the type of index. Based on, the method to be included in the returned list.
[Aspect 9]
A method according to aspect 1, wherein the predetermined first threshold is defined so that the method ends in response to a determination that the number of search results falls below a certain number.
[Aspect 10]
A method of the method of aspect 1, wherein the query is updated after the process of returning the result tile of the first list.
[Aspect 11]
In the method according to aspect 1, further
The process of excluding results from at least one of the result tiles of the first list and the result tiles of the second list before processing the result tiles of each list.
Including methods.
[Aspect 12]
The method of aspect 1, wherein the query comprises a quality threshold level.
[Aspect 13]
The method of aspect 1, wherein the query includes a time limit, and the search is performed with this time limit.
[Aspect 14]
A way to perform a recursive search of tile sets based on query tiles,
For each tile in the tile set, until the result set is complete
Extracting a set of tiles from the next level, and adding the next level tile set to the result set,
And the process of executing
In response to the determination that the magnification level is a predetermined target level, the process of evaluating the quality of matches in the result set, and
For each tile in the result set, in response to the determination that the magnification level is below the target level.
Adding tiles to a subset in response to a determination that the number of tiles in the subset is greater than or equal to a third threshold.
In response to the determination that the number of tiles in the subset is less than the third threshold,
Recursively searching the subset
Adding the results of this recursive search to the tentative result set,
Clearing the above subset and
To do,
Recursively searching the subset,
Adding the search results to the provisional result set,
Clearing the subset and returning the provisional result set,
And the process of executing
Including methods.
[Aspect 15]
In the method according to aspect 14, the process of evaluating the quality of the match determines whether or not the match value of the first tile in the result set as compared with the query tile is less than a predetermined value. A method that has to do.
[Aspect 16]
In the method of aspect 14, each pixel in a tile has at least one numerical value representing at least one of the color and brightness of each pixel, and each tile is in that tile. A method having said at least one numerical vector for every pixel.
[Aspect 17]
A method of continuously processing an image data repository performed by a computer.
The process of receiving a query specification containing a request for data, and
The process of receiving the system designation of the computer on which the method is executed, and
The process of comparing the query specification with the system specification to determine the domain specification,
The process of initiating a query for the repository based on the domain designation, and
The process of receiving the result of the query including the image data,
The process of displaying the interactive and iterative search of the resulting image data on the graphical user interface,
The process of receiving the input of the result image data via the graphical user interface and
The process of updating the query based on the input received, and
The process of displaying the updated graphical user interface based on the updated result image data, and
Including methods.
[Aspect 18]
A method according to aspect 17, wherein the repository of image data comprises digital data of a microscope.
[Aspect 19]
In the method of aspect 17, the continuous processing of the image data is a method of incrementally producing results so that the results are available before the processing is completely completed.
[Aspect 20]
A method in which the query designation implicitly defines indexing and transformation of data in the method of aspect 17.

Claims (15)

A computer-implemented method of searching a database of images based on query tiles.
In response to the determination that the magnification level of the query tile is larger than the first threshold value, the image stored in the database is searched, and the first magnification level of the query tile satisfies the query tile. The process of returning the result tile of the list and
In response to the determination that the magnification level of the query tile is equal to or less than the first threshold value, the image stored in the database is searched, and the tile having the next lowest magnification level after the magnification level of the query tile is selected. The process of retrieving and returning the result tile of the second list that fills the query tile at the next lowest magnification level,
In the process of processing the result tiles of each list, for each of the result tiles
Adding the result tile to a subset of the result tiles of multiple subsets,
Recursively searching the subset for the query tile in response to a determination that the total number of result tiles in the subset is greater than or equal to the second threshold.
Saving the results of each recursive search for the subset into the remaining subsets of the subset.
Recursively searching the remaining subset for the query tile and saving the results of the search for the remaining subset.
Have a process and
Including methods. The method of claim 1, wherein each magnification level corresponds to one level in the quadtree, and each level in the quadtree comprises a result tile representing an image. The method of claim 2, wherein the child tile has an association with the parent tile. The method of claim 3, wherein the parent tile is at least one of the downsampled tiles and the lowpass spatially filtered tiles of the corresponding at least one child tile. The method of claim 2, wherein the process of removing the next lowest magnification level tile is to remove the child of the quadtree. In the method of claim 1, in response to the determination that the total number of result tiles in the subset is less than the second threshold.
Recursively search the subset for the query tile
Add recursive search results to the tentative result set and
Clear the subset and then
Recursively search the subset for the query tile
Add the search results to the provisional result set and
Clear the subset and
A method of returning the provisional result set. The method of claim 1, wherein the query includes an image and the magnification level of the query is the magnification level of the image. In the method of claim 7, the result tile is at least one of the magnification of the result tile, the image of the query, the filename of the index associated with the result tile, the size of the result, and the type of index. How to be included in the returned list based on. The method of claim 1, wherein the predetermined second threshold is defined so that the method terminates in response to a determination that the number of search results falls below a certain number. The method of claim 1, wherein the query is updated after the process of returning a tentative result set. The method of claim 1, wherein the query comprises a quality threshold level. The method of claim 1, wherein the query includes a time limit and the search is performed with this time limit. A computer-implemented way to perform a recursive search of tile sets based on query tiles,
For each tile in the current tile set, until the result set for the query tile is complete
Extracting a set of tiles from the next level of the current tile set, and adding the next level tile set to the result set,
And the process of executing
A process of evaluating the degree of matching of the result set with respect to the query tile in response to the determination that the magnification level of the query tile is a predetermined target level.
For each tile in the result set, in response to the determination that the magnification level of the query tile is below the target level.
Adding tiles to a subset in response to a determination that the number of tiles in the subset is greater than or equal to a third threshold.
In response to the determination that the number of tiles in the subset is less than the third threshold,
Recursively searching the subset for the query tile
Adding the results of this recursive search to the tentative result set,
Clearing the above subset and
To do,
Recursively searching the subset for the query tile,
Adding the search results to the provisional result set,
Clearing the subset and returning the provisional result set,
And the process of executing
Including methods. In the method according to claim 13, in the process of evaluating the degree of matching with respect to the query tile, the match value of the first tile in the result set as compared with the query tile is less than a predetermined value. A method having to determine if there is. In the method of claim 13, each pixel in a tile has at least one numerical value representing at least one of the color and brightness of each pixel, and each tile is that tile. A method having the vector of at least one numerical value for all pixels in.

Images