WO2023210185A1

WO2023210185A1 - Microscope image information processing method, microscope image information processing system, and computer program

Info

Publication number: WO2023210185A1
Application number: PCT/JP2023/009276
Authority: WO
Inventors: 宣仁森; 泰之木田
Original assignee: 国立研究開発法人産業技術総合研究所
Priority date: 2022-04-26
Filing date: 2023-03-10
Publication date: 2023-11-02

Abstract

Provided is a microscope image information processing method that avoids re-imaging of a sample and can be implemented by an existing computer system.　In a microscope image information processing method, a captured image (50) of a portion of a sample observed using a microscope is acquired and stored in a storage region (32) (ST1), when it is detected that the captured image (50) is stored in the storage region (32), feature point information which is information relating to a feature point in the captured image (50) is calculated, feature point information of a previous captured image and the feature point information of the captured image (50) are used (ST2) to execute matching processing of the feature point of the previous captured image and the feature point of the captured image (50), joining processing is executed on the basis of the result of the matching processing and a plurality of captured images (50) stored in the storage region (32) up to the present time to generate a joined image (52) (ST4), the joined image (52) is output for display, and the processing described above is repeated until imaging of the portion of the sample is terminated.

Description

Microscope image information processing method, microscope image information processing system, and computer program

The present invention relates to a method of processing microscopic image information.

Conventionally, a technology called virtual slide exists as a technology for creating high-definition, large-area digital images of glass slide specimens and the like observed under a microscope. Since virtual slides are image data, they are easier to handle than storing the slide glass specimens themselves. Virtual slides can be used, for example, for remote pathology diagnosis and digital storage of pathology samples.

There are several methods for generating virtual slides, such as Off-line stitching and Real-time stitching (for example, Non-Patent Document 1). Off-line stitching is a method in which multiple images of a slide glass specimen or the like are taken over a wide area, and then the multiple images are stitched offline to generate a single image. Real time stitching is a method of stitching together multiple captured images to create a single image while observing a glass slide specimen. Additionally, as a method of creating virtual slides, there is also a device called a slide scanner that automatically scans a glass slide specimen.

However, in offline stitching, it is not possible to preview and check the images being stitched while photographing slide glass specimens. There has been a problem in that re-photographing work is likely to occur due to reasons such as missing images. In addition, real time stitching requires dedicated software and a dedicated camera compatible with the software, and cannot be used with a general digital camera. Therefore, it has been difficult to flexibly change the photographing conditions during photographing. Additionally, a specialized automated device called a slide scanner for generating virtual slides is generally very expensive, and there are limited environments in which this device can be introduced.

The present invention has been made in view of the above problems.

In order to solve the above problems, one aspect of the present invention is a microscope image information processing method executed by a computer system, which acquires a captured image of a portion of a sample observed using a microscope and stores it in a storage area. a first step of storing the captured image; and a second step of calculating feature point information, which is information regarding the feature points of the captured image, when it is detected that the captured image has been saved in the storage area; a third step of executing a matching process between the feature points of the previous captured image and the feature points of the captured image using the feature point information of the image and the feature point information of the captured image; a fourth step of generating a joined image by performing a joining process based on the result of the matching process and the plurality of captured images that have been saved in the storage area to date; and a fifth step of displaying and outputting an image, and repeating the first step to the fifth step until the photographing of a portion of the sample is completed.

In another aspect of the present invention, the first step of acquiring a photographed image of a part of a sample observed using a microscope and storing it in a storage area is performed, and the photographed image is stored in the storage area. When it is detected, a second step of calculating feature point information, which is information regarding the feature points of the captured image, is executed, and the feature point information of the previously captured image and the feature point information of the captured image are calculated. and a third step of executing a matching process between the feature point of the previous captured image and the feature point of the captured image using a fourth step of performing a joining process to generate a joined image based on the plurality of captured images stored in a region; and a fifth step of displaying and outputting the joined image. , is a microscope image information processing system that repeats the first step to the fifth step until the photographing of a portion of the sample is completed.

Another aspect of the present invention is a computer program that causes a computer system to execute the above-described microscope image information processing method.

FIG. 3 is a diagram illustrating an overview of sequential joining processing in a microscope image information processing method according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an overview of overall configuration processing in a microscope image information processing method according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of the results of matching a plurality of feature points detected in the Nth image with a plurality of feature points detected in the immediately previous merged image. FIG. 7 is a diagram conceptually explaining a process of recalculating a global transformation matrix using a connected undirected graph. FIG. 7 is a diagram conceptually explaining a process of recalculating a global transformation matrix using a connected undirected graph. It is a figure which shows an example of the flowchart of the sequential joining process in the microscope image information processing method based on one Embodiment of this invention. FIG. 7 is a diagram showing an example (partially) of a spliced image (without distortion correction) generated by sequential splicing processing. FIG. 7 is a diagram illustrating an example (with distortion correction) of a spliced image (part) generated by sequential splicing processing. FIG. 7 is a diagram illustrating an example of a flowchart of processing in step S120 (processing for the image file from the second time onwards). FIG. 3 is a diagram showing an example of a flowchart of overall configuration processing in a microscope image information processing method according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an example of a spliced image (partially) that has been subjected to bundle adjustment (distortion correction) in the overall configuration process. It is a figure which shows an example of the image produced|generated by joining all images. FIG. 3 is a diagram illustrating an example in which a rectangular area is divided into multiple tiles. FIG. 7 is a diagram illustrating the effect of additional matching processing of region overlapping images in sequential joining processing. FIG. 7 is a diagram illustrating the effect of additional matching processing of region overlapping images in sequential joining processing. FIG. 7 is a diagram illustrating the effect of reducing misalignment between images by calculating Max Spanning Tree and simple bundle adjustment in sequential splicing processing. FIG. 7 is a diagram illustrating the effect of reducing misalignment between images by calculating Max Spanning Tree and simple bundle adjustment in sequential splicing processing. FIG. 7 is a diagram showing the effect of distortion correction in overall configuration processing. FIG. 7 is a diagram showing the effect of distortion correction in overall configuration processing. 3 is a diagram showing an example of a hardware configuration of a computer device 30. FIG.

Embodiments of the present invention will be described in detail below with reference to the drawings.
(Overview of microscope image information processing method)
FIGS. 1 and 2 are diagrams illustrating an overview of the microscope image information processing method according to the present embodiment. The microscope image information processing method according to the present embodiment can be executed by a microscope image information processing system 1 that includes an existing general microscope 10, a camera 20, and a computer device 30. The microscope image information processing method according to this embodiment is roughly divided into two phrases: sequential joining processing and overall configuration processing. The sequential bonding process is a process that generates the latest bonded image every time an image of a part of the slide glass specimen is taken, and finally generates a bonded image of the entire slide glass specimen. Further, the overall configuration process is a process that is executed after the photographing of the slide glass specimen is completed, and is a process that is executed on the joined images that have been sequentially subjected to the joining process. The overall configuration processing mainly performs various processing to improve the quality of the spliced images and saves the processed images. In sequential joining processing, processing is mainly executed with a small waiting time, and in overall configuration processing, processing is mainly executed that requires time or calculation cost.

FIG. 1 is a diagram illustrating an overview of sequential joining processing. While observing the glass slide specimen through the microscope 10, the user uses the camera software 40 to sequentially photograph each part of the glass slide specimen with the camera 20. The photographed image 50 of each part of the slide glass specimen is written to the hard disk drive (HDD) 32 included in the computer device 30 by the camera software 40 each time the photograph is taken (ST1). Furthermore, the HDD 32 is monitored by the joining software 42 (ST2), and each time it is detected that an image file 50 has been written to the HDD 32, the written image file 50 is read from the HDD 32 (ST3). , a joining process is executed (ST4). Each time an image 50 of each part of the slide glass specimen is photographed and stored in the HDD 32, the photographed image is read out from the HDD 32 and the joining process is performed. This process is repeated, and each time the image is taken, a joined image is created. 52 will be updated. Then, each time the spliced image 52 is updated, the updated spliced image 52 is displayed as a preview to the user on a display device such as a display. In the microscope image information processing method according to the present embodiment, the merged image 52 that is currently being generated is previewed by the user at any time, so the user can confirm at any time that the appropriate merged image 52 is being generated. It is. As a result, it is possible to reduce the possibility that re-photographing work will occur after the photographing is completed, as in the case of virtual slide generation using conventional off-line stitching.

More specifically, the sequential joining process can be executed as follows. That is, matching of the feature points between the previously captured image and stored in the HDD 32 and the newly captured image 50 captured and stored in the HDD 32 is performed, and a transformation matrix between the two images is calculated. Furthermore, based on the results, feature point matching with another image that is already stored in the HDD 32 and whose movement destination area overlaps is executed, and a transformation matrix is calculated. The user's waiting time can be reduced by limiting the calculation target to only images with overlapping movement destination areas, rather than all stored captured images. Furthermore, by calculating the Max Spanning Tree and performing simple bundle adjustment as necessary, it is possible to reduce the accumulation of misalignment between images.

FIG. 2 is a diagram illustrating an overview of the overall configuration processing. For example, when the user inputs that the sequential joining process has been completed, the joining software 42 executes processing to improve the quality of the images (ST5), and stores the joined images 52 after performing the processing on the HDD 32. Save (ST6).

More specifically, the overall configuration process may be executed as follows, for example.
(1) Bundle adjustment (calculation of a transformation matrix that minimizes the deviation of all feature point pairs and distortion correction of the lens of the camera 20)
(2) Seam calculation (search for the best break between images)
(3) Exposure correction (correction of exposure time and vignetting between images)
(4) Blending processing (combining images so that the breaks between images are less noticeable)
(5) Writing the overall configuration processed image to the HDD 32 of the computer device 30 A part of the above processing is performed by dividing the entire image into small areas (tiles). In this case, (2) to (5) or (3) to (5) may be processed for each tile. This makes it possible to reduce the amount of memory and calculation time used for one-time processing.

Note that in the description of this embodiment, a virtual slide generated by imaging a slide glass specimen will be described as an example, but this is just an example. The microscope image information processing method according to the present embodiment is similarly applicable to, for example, imaging cells cultured in a petri dish.

The sequential joining process and the overall configuration process will be described in more detail below.
(Sequential joining process)
Referring to FIG. 3, in order to perform the (N+1)th sequential joining (N: an integer greater than or equal to 1), the photographed image photographed the Nth time (hereinafter referred to as the "Last (N) image" for convenience of explanation) 50' and the (N+1)th photographed image (hereinafter referred to as "New (N+1) image" for convenience of explanation) 50 newly photographed and stored in the HDD 32, a plurality of characteristics are determined using a known method. Detection of points, calculation of feature amounts, and matching of detected feature points are performed. FIG. 3 is a diagram showing an example of the result of matching a plurality of feature points detected in the Last (N) image 50' and a plurality of feature points detected in the New (N+1) image 50. Note that in FIG. 3, for convenience of explanation, only some of the matching results are shown by broken lines. Next, a transformation matrix R _N,N+1 is calculated between the Last (N) image 50' and the New (N+1) image 50 based on the matching result. The transformation matrix R _N,N+1 is a matrix (affine transformation matrix) indicating the movement distance and rotation amount between the Last (N) image 50' and the New (N+1) image 50. That is, the following relational expression holds true.

Here, the matrix

is a matrix representing the feature points of the New (N+1) image 50. Also, the matrix

is a matrix representing the feature points of the Last(N) image 50'. The calculated transformation matrices R _N,N+1 are sequentially stored in a storage area such as the HDD 32 of the computer device 30 in the form of an array or the like. Furthermore, the degrees of freedom of the components a, b, c, d, e, and f of the transformation matrix R _N,N+1 may be lowered depending on the performance of the microscope. For example, if the optical system and operating parts of a microscope have sufficient precision, the relationship between the captured images can be considered as a parallel movement, so a = e = 1, b = d = 0, and only c and f are variables. Good (2 degrees of freedom). In addition, when it is necessary to consider not only parallel movement but also rotation, such as when the accuracy is slightly poor or the slide glass specimen is not fixed loosely, use a = cos θ, b = sin θ, d = -sin θ, and e = cos θ. θ, c, and f may be variables (3 degrees of freedom). If the accuracy is poor or the lens magnification changes during shooting, set t, θ, c, and f as variables such as a=t・cosθ, b=sinθ, d=−sinθ, and e=t・cosθ. (4 degrees of freedom). By setting appropriate degrees of freedom according to the microscope and specimen, it is expected that the calculation time and memory consumption required for joining can be reduced, and the accuracy of joining can be improved.

Next, a global transformation matrix for the New (N+1) image 50 is calculated. The "global transformation matrix" is information representing how much the New (N+1) image 50 has moved and rotated from a specific image with respect to the specific image. Here, if the global transformation matrix of the New (N+1) image 50 is R _N+1 with the first captured image as a reference, the following relationship holds true.

That is, the global transformation matrix R _N+1 of the New (N+1) image 50 is each transformation matrix between the immediately previous captured image and the new captured image, which is calculated in each joining process before the Nth sequential joining. It can be calculated as the product of R _1,2 , R _2,3 , . . . R _N,N+1 . Using the calculated global transformation matrix R _N+1 of the New (N+1) image 50, the image data of a specific reference image (here, the first captured image) and the image data of the New (N+1) image 50 are joined. Processing is performed and a spliced image 52 is generated.

By the way, it is possible to calculate the global transformation matrix R _N+1 by the method described above, but in order to calculate R _N+1 , a plurality of matrices R _1,2 , R _2,3 , . . . R _N,N+1 are required. , the errors occurring in each matrix will accumulate. In this embodiment, the following processing was further performed in consideration of this point.

That is, by calculating the global transformation matrix R _N+1 , the rough destination (hereinafter referred to as "movement destination area") of the New (N+1) image 50 from the reference image (first image) can be found. Then, by comparing the movement destination area of the New (N+1) image with the movement destination area of each image calculated in the past, in addition to the Last (N) image 50', the New (N+1) image 50 and the movement destination area of each image calculated in the past are compared. Search for past merged images with overlapping destination areas. Here, "movement destination areas overlap" may be such that, for example, it is determined that the movement destination areas overlap when the overlapping area between them exceeds "0", or a predetermined value may be used. If the overlapping area between the destination areas exceeds the threshold value, it may be determined that the destination areas overlap. If it is determined that the K-th (0<K<N) joined image 50' overlaps the New (N+1) image 50 in the movement destination area, then the New (N+1) image 50 and the K-th joined image 50', and a transformation matrix R _K,N+1 between the K-th spliced image 50' and the New (N+1) image 50 is calculated. The above-mentioned series of processing is called "additional matching processing for region overlapping images." Also, based on this information, a connected undirected graph with each image as a vertex is calculated.

FIGS. 4A and 4B are diagrams conceptually explaining the recalculation process of the global transformation matrix using the graph. Before recalculating the global transformation matrix, as shown in FIG. 4A, the first image, the second image, the third image, the Kth image, the Nth image, and the ) Image 50) A path was determined to follow each image in order. By performing the additional matching process for region overlapping images, the K-th image and the N+1-th image are linked, for example, as shown in FIG. 4B. Then, a transformation matrix R _{K,N+1 between the K-th image and the N+1-th image is calculated based on the result of the matching process between the feature points of the K-th image and the feature points of the N+1-th} image. . Further, the optimal path for tracing the entire image can be determined by calculating the Max Spanning Tree, for example. Max Spanning Tree is a spanning tree with the maximum sum of weights in a weighted connected undirected graph, and it is known that it can be calculated using algorithms such as Kruskal's method. In the present invention, the number of feature points matched between each image can be used as the weight of the graph, but another index may be used as the weight. After calculating the Max Spanning Tree, the center of the tree is used as a new reference. Through this process, it is better to use the image near the center of the image as a reference, for example, the third image in FIG. 4B, rather than the path that sequentially follows from the first image to the N+1-th image as shown in FIG. 4A. Since the overall route is shorter (fewer matrices are multiplied and errors are reduced), the third image can be determined as the new reference image. In this case, the following relationship holds true.

By performing sequential joining processing on the N+1-th captured image using the global transformation matrix R _N+1 calculated as described above, errors can be further reduced.
(Overall configuration processing)
The overall configuration process is a process that is executed after all the images of the slide glass specimens are completed, and the overall configuration process improves the quality of the bonded image. More specifically, the following processes are executed in the overall configuration process in this embodiment.
(1) Bundle adjustment (calculation of a transformation matrix that minimizes the deviation of all feature point pairs and distortion correction of the lens of the camera 20)
(2) Seam calculation (search for the best break between images)
(3) Exposure correction (correction of exposure time and vignetting between images)
(4) Blending processing (combining images so that the breaks between images are less noticeable)
(5) Writing the overall configuration processed image to the HDD 32 of the computer device 30 A part of the above processing is performed by dividing the entire image into small areas (tiles). In this case, (2) to (5) or (3) to (5) may be processed for each tile. This makes it possible to reduce the amount of memory and calculation time used for one-time processing.
(bundle adjustment)
As described above, matching of feature points is performed in the sequential joining process. When the feature points are moved and rotated as indicated by the transformation matrix, the total error is calculated by calculating the amount of error between matched feature points (reprojection error) for all feature points. It can be done. By minimizing this overall error using the least squares method, the sequentially joined images can be made into images with better quality as a whole. As an example, the Levenberg-Marquardt method can be used. The Levenberg-Marquardt method is one of the methods for solving nonlinear least squares problems.

Here, in this embodiment,
x _i : Component of the transformation matrix of the image at processing time i J _i : Jacobian at processing time i (T on the right shoulder indicates transposition)
r _i : Error λ between feature points at processing time i: Real number I greater than or equal to zero that is adjusted according to the size of the error: Equation (1) can be calculated as a unit matrix.

According to the above equation (1), the parameter J and the parameter r are calculated for the parameter x at a certain processing point. Furthermore, using the calculated parameters J and r, parameters J and r are further calculated for the next parameter x. By repeating this process, a global transformation matrix that minimizes the reprojection error between all matched feature points is calculated, and bundle adjustment can be performed based on this.
(Processing flow: sequential joining process)
FIG. 5 is a diagram showing an example of a flowchart of sequential joining processing.

First, input from the user of the microscope image information processing system 1 to designate a folder to be monitored on the HDD 32 is received via an application such as camera software 40 or joining software 42 running on the computer device 30 (step S102). The joining software 42 checks for updates to the designated folder at regular intervals (step S104). Unless the joining software 4 receives an input indicating the end of the sequential joining process (step S106: No), the sequential joining process is continued and the process proceeds to step S108.

In step S108, if the joining software 42 determines that the image file has not been updated for the folder confirmed in step S104 (step S108: No), the process returns to step S104. If the bonding software 42 determines that the image file has been updated, that is, if a new image of the slide glass specimen has been taken and saved in the folder (step S108: Yes), the image file has been updated. It is determined whether this is the first storage, that is, the first storage of the captured image of the slide glass specimen (step S110). If the image file is not updated for the first time (step S110: No), processing for the second and subsequent saved image files is executed (step S120). The process will be described in detail later.

If the update of the image file is the first saving of the image file (step S110: Yes), the saved image file is read from the HDD 32, feature points and feature amounts in the image file are calculated, and the calculation result is are stored in an array for storing information on feature points and feature amounts of each image (step S112). At this time, if the distortion parameters, vignetting parameters, etc. are known in advance, correction may be performed when reading the image file. Distortion may occur in the captured image due to the optical system lens of the camera 20. It is generally known that the transformation from "true coordinates x, y, z without distortion" to "coordinates u, v after photographing with distortion" can be calculated, for example, by the following formula. (For example, Zhengyou Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330-1334, 2000. https://www.microsoft.com/en-us/research /wp-content/uploads/2016/02/tr98-71.pdf)

Here, k _n , p _n , c _x , c _y , f _x , and f _y are distortion parameters, and these values are estimated before performing distortion correction. More specifically, before performing the distortion correction, a grid pattern in which squares of the same size are lined up is photographed using the same camera 20 that photographs the slide glass specimen. Then, the values of the parameters k _n , p _n , c _x , c _y , f _x , and f _y can be estimated from the distortion of the photographed image. Using the estimated parameter values, it is possible to correct the distortion of the captured image by inversely transforming (u, v) → (x, y) (here, the Z coordinate is ignored since it is a two-dimensional system). can. Note that since there is no analytical inverse function, inverse transformation can be performed using an algorithm that calculates an approximate solution, such as Newton's method. Further, only some parameters may be considered (for example, only the value of the parameter k ₁ and the value of the parameter p ₁ are considered, and the values of other parameters are regarded as "0", etc.). Further, distortion correction may be performed using other distortion models.

As described above, by correcting the distortion of the photographed images and generating a spliced image using each photographed image after distortion correction, it is possible to reduce the deviation in the spliced images. FIGS. 6A and 6B are diagrams showing an example of a joined image (partially) generated by the inventors of the present application using the microscope image information processing method according to the present embodiment. Figure 6A shows the merged images obtained by sequentially joining the captured images without performing distortion correction, and Figure 6B shows the merged images obtained by sequentially joining the captured images after performing distortion correction. Show images. The error for the stitched image in FIG. 6A was about 3.32e+05, and the error for the stitched image in FIG. 6B was about 1.57e+04. Errors were significantly reduced by applying distortion correction to each captured image and sequentially combining them. Note that the error (reprojection error) was calculated using the following procedure. That is, (1) the coordinates of the feature points of each image in the reference coordinate system (the coordinate system of the reference image) are calculated using the global transformation matrix. (2) Calculate the deviation in the reference coordinate system between the matched feature points. Ideally, if the matched feature points were transferred to the reference coordinate system, they would overlap exactly, but due to distortions, etc., this does not happen. This becomes an error.

Returning to FIG. 5, next, the image file is reduced at a predetermined ratio and stored in an array for saving image data (step S114). By retaining the reduced image in the RAM 33, HDD 32, etc., memory consumption and storage capacity consumption can be reduced compared to having the original size image. However, full-size image data may be stored in the array without reducing the image file, and from now on, the spliced image may be generated and treated as a full-size image rather than a reduced image.

The global transformation matrix (here, the unit matrix) of the first captured image reduced in step S114 is stored in an array for saving the global transformation matrix (step S116). A preview image is synthesized using the reduced image data saved in step S114 and the global transformation matrix (unit matrix) saved in step S116, and the preview image is displayed. Note that a preview of a reduced image of the first photographed image may be displayed (step S118).

The above process is performed in step S106 when the joining software 42 receives a sequential request from the user via an input device such as a keyboard or a mouse due to reasons such as the completion of the sequential joining process (that is, the completion of virtual slide generation). The process continues until an input indicating the end of the joining process is received (step S106: Yes). When receiving an input from the user to end the sequential joining process, the sequential joining process is ended and the process proceeds to the overall configuration process.
(Processing flow: Processing of step S120)
FIG. 7 is a diagram illustrating an example of a flowchart of the process in step S120 (processing for the image file from the second time onward).

First, image n, which is a partial image of a slide glass specimen stored in the HDD 32, is read for the nth time (n: an integer of 2 or more), and feature points and feature amounts are calculated (step S1202). At this time, similar to step S112 in FIG. 5, if the distortion parameters, vignetting parameters, etc. are known in advance, correction may be performed when reading the image file. Next, feature point matching is performed between image n and image n-1, which is the image taken the (n-1)th time, and a transformation matrix (hereinafter referred to as "image pair matrix") between both images is performed. ) is calculated (step S1204). If the process in step S1204 is successful (step S1206: Yes), the destination area of image n is calculated, and based on this calculation result, one or A plurality of images (hereinafter also referred to as "paired images"; image K in the example of FIG. 4B corresponds to this) are selected (step S1208). Next, matching of feature points is performed between image n and all paired images selected in step S1208, and each image pair matrix (transformation matrix R _K,N+1 in the example of FIG. 4B) is calculated (step S1210). The matching of feature points executed in step S1210 is a process with high calculation cost, and if the matching of feature points is performed for all saved images, the user's waiting time will increase. Therefore, it is possible to reduce the user's waiting time by performing feature point matching only on images in which the movement destination areas selected in step S1208 overlap.

Next, the feature points and feature amounts of image n are stored in an array (step S1212). The image n is reduced by a predetermined ratio, and the image data of the reduced image is stored in an array (step S1214). The image pair matrix obtained in step S1210 is stored in an array (step S1216). A global transformation matrix is calculated from the image pair matrix stored in step S1216 and stored in an array (step S1218). Note that the global matrix of image n is the image pair matrix between image n and the paired image of image n (the image selected in step S1208, image K in the example of FIG. 4B), and the global matrix of the paired image of image n. It can be calculated as a product with a transformation matrix. Further, in step S1218, the global transformation matrix of all the saved images may be recalculated as necessary. "As needed" may include, for example, "every predetermined number of times,""when input is received from the user,""when the reprojection error between all matched feature points reaches a certain value," and the like. Furthermore, when recalculating the global transformation matrix for all images, a Max Spanning Tree may be calculated with the number of feature matchings as weights. Set the image at the center of the Tree (the third image in the example of Figure 4B) as the new reference image (the global transformation matrix of the third image becomes the identity matrix), and calculate the product of the image pair matrices in order of the edges of the Tree. This allows us to calculate the global matrix for each image. Additionally, in addition to these, bundle adjustment may be performed a small number of times (for example, twice) to calculate the global transformation matrix. Bundle adjustment can be performed by minimizing the reprojection error between all matched feature points, for example by the Levenberg-Marquardt method. Furthermore, the calculated image pair matrix may be updated based on the recalculation results of the global transformation matrices of all images (for example, if the global transformation matrices of image A and image B are Ma and Mb, respectively, image A The image pair matrix between and image B can be calculated as the product of matrix Ma and matrix Mb ⁻¹ ). This allows the next recalculation to be started from a state with fewer errors.

A preview image is generated using the reduced image data saved in step S1214 and displayed on a display device such as a display (step S1220). Here, since the preview images up to image n-1 have been combined, the preview image may be generated by joining the reduced-sized image n to the preview image of image n-1. Furthermore, if the global transformation matrix is recalculated for all of the saved images in step S1218, the preview images generated so far may be discarded and preview images may be generated. After this, the process transitions to step S104 in FIG.

In addition, in step S1206, if the process in step S1204 is not successful (step S1206: No), matching of feature points is performed between image n and all saved images, and an image pair matrix is calculated (step S1222). As a result, if at least one pair of images (a pair of one or more images with overlapping destination areas) exists (step S1224: Yes), the process transitions to step S1212. If the image pair does not exist (step S1224: No), the process transitions to step S104 in FIG. 5.

Note that if the processing in step S1204 is not successful (step S1206: No), that is, if the matching of feature points between image n and image n-1 fails, for example, the photographing of the slide glass specimen is This can occur when restarting from a different field of view that is significantly different from the one that was being photographed (because the entire area of the slide glass specimen is not necessarily photographed by tracing it with one stroke). Further, the process in step S1222 may be terminated when at least one image pair matrix has been calculated. In that case, the subsequent processing moves to step S1208.

In addition, the array used in the flows of FIGS. 5 to 7 includes an array in which image pair matrices are stored, an array in which reduced images are stored, an array in which global transformation matrices are stored, and feature points and feature amounts of each image. The data in the array where is saved is carried over to the overall configuration processing described later. Furthermore, by saving all or part of this data as a file in a storage area such as the HDD 32 of the computer device 30, the overall configuration process can be suspended and the overall configuration process performed at a later date. It can be executed on another computer device, etc.
(Processing flow: Overall configuration processing)
FIG. 8 is a diagram illustrating an example of a flowchart of the overall configuration process.

Bundle adjustment is performed on all images on which the sequential splicing process has been performed. As described above, the bundle adjustment may be performed by the Levenberg-Marquardt method (the above equation (1)) or the like (step S302). Here, if the distortion parameters of the optical system lens of the camera 20 are not known in advance, in addition to the global transformation matrix, distortion parameters common to all images are included as adjustment target parameters for bundle adjustment, and the distortion parameters of the feature point coordinates are included. By minimizing the reprojection error calculated after correction, it is possible to correct the optical distortion of the camera 20. That is, at this time, the parameter x in the above equation (1) includes the components of the transformation matrix of all images and the common distortion parameter. That is, if distortion is not taken into consideration, the parameter x _i in equation (1) is such that the components (a, b, c, d) of the global transformation matrix of each image are arranged (x _i =(R ₁ , R ₂ , . . . ., R _n ), where (R ₁ =a ₁ , b ₁ , c ₁ , d ₁ )). _When considering distortion, _the _distortion _parameter _of _equation ( ₂ ) _is It becomes the form by adding . This becomes a common distortion parameter. By calculating the error r using this parameter x _i , it is possible to calculate an improved x _(i+1) based on equation (1), and by repeating this, an even more improved global transformation matrix and distortion can be calculated. Parameters are obtained. As initial values, k ₁ , p ₁ , . . . are determined by tentatively setting no distortion (k ₁ =p ₁ = . . . =0) and repeatedly calculating equation (1). It is also possible to calculate by assuming distortion parameters specific to each image, but by using common distortion parameters, calculation time can be further reduced, and the location where the slide glass specimen is observed can be changed. This is sufficiently practical if the lens of the camera 20 is not changed. Note that since the distortion parameters are calculated using only the feature point coordinates in step S302, it is the subsequent processing that actually performs distortion correction targeting all pixels of each image.

Next, distortion correction is performed on the reduced image (the reduced image stored in the array in steps S114 and S1214) (step S304). Note that if the distortion parameters are known in advance prior to the sequential splicing process and distortion correction has already been performed for each captured image in step S112 in FIG. 5 and step S1202 in FIG. 7, the correction in this step is omitted. be done. FIG. 9 is an example of a spliced image obtained by performing distortion correction on the same captured images as in FIGS. 6A and 6B by including a common distortion parameter in the parameter x in equation (1), and then splicing the images. FIG. The error for the stitched image in FIG. 6A was about 3.32e+05, and the error for the stitched image in FIG. 6B was about 1.57e+04. The error of the spliced image in FIG. 9 was approximately 1.55e+03. It can be seen that the error in the spliced image in FIG. 9 is further reduced compared to the spliced image shown in FIG. 6B, in which the captured images are subjected to distortion correction in advance and spliced sequentially.

Next, a seam between the photographed images is calculated using the reduced image (step S306). By performing seam calculations, images can be joined at appropriate breaks. As a method for calculating the seam, for example, existing methods such as a method using a Voronoi diagram, dynamic programming, and graph cut can be used. Note that seam calculation takes time, so in this embodiment, seam calculation is performed in overall configuration processing rather than sequential joining processing so that the user's waiting time can be long. There is. Further, if a full-size image is used, it takes time to calculate the seam, but in this embodiment, a reduced image is used as described above, which contributes to shortening the calculation time.

Next, the size (width and height) of the image generated by joining all the captured images is calculated. FIG. 10 is a diagram showing an example of an image generated by joining all images. Note that FIG. 10 is shown in a simplified manner for convenience of explanation. Assuming that there are six images (images 1 to 6), the image generated by joining images 1 to 6 is image 60, which is configured to include all of images 1 to 6. In this step, the size (width w, height h) of the image 60 is calculated as follows. That is, (1) using the global transformation matrix and distortion correction parameters obtained in step S302, calculate the movement destination area (in the reference coordinate system) of each image 1 to 6 (the size of each image 1 to 6). (2) Calculate a rectangular area that includes the entire destination area obtained in (1). Further, the calculated area is divided into a plurality of small areas (hereinafter referred to as "tiles", "tile images", etc.) (step S308). FIG. 11 is a diagram showing an example in which the rectangular area 60 is divided into a plurality of (here, two for simplicity of explanation) tiles 65 and 66. Also, images that intersect with the tiles 65 and 66 are defined as a tile image set. In this example, the tile image set for tile 65 is images 1, 2, 3, 4, 5, and the tile image set for tile 66 is images 4, 5, 6. For example, when tile 65 is joined, an image larger than the size of tile 65 is generated, but the protruding portion is cut off. Further, the size and shape of the tile can be determined as follows. That is, (1) a rectangle with a predetermined size (1024 pixels x 1024 pixels, etc.), (2) the user is required to specify it in a configuration file, etc., (3) it is automatically determined by taking into account the memory of the computer device 30, etc. It is fine. Regarding (3), for example, the memory required to process one tile can be estimated by "the number of images included in the tile image set x the size of one full-size image", so this required memory amount is The size of the tiles may be determined so that the software does not exceed 10% of the available memory, and so on. Further, the shape of the tile may be a specific predetermined shape (such as a square), or may be a shape similar to each of the images 1 to 6 (such as a rectangle with an aspect ratio of 3:4). Hereinafter, in the processing of steps S310 to S318, full-size images are used to join the images. Furthermore, if a full-size image is all processed at once, the amount of memory used and the calculation time will increase, so the entire image is divided into tiles and the processes of steps S310 to S318 are executed for each tile.

First, in this step, an image where the movement destination area intersects with the tile to be processed is identified. In the example of FIG. 11, the tile image set of tile 65 is images 1, 2, 3, 4, and 5, and the tile image set of tile 66 is images 4, 5, and 6. (Step S310). (The specified set of images is referred to as a "tile image set.") Next, each image file of the tile image set specified in step S310 is read from the HDD 32 (step S312). Regarding the tile image set read out in step S312, the distortion parameter (either a distortion parameter known in advance or a distortion parameter estimated by including a common distortion parameter in the parameter x of the above equation (1) may be used. ) is performed, and if the vignetting parameter is known in advance, image correction is performed based on the vignetting parameter. If the vignetting parameters are not known in advance, exposure correction is performed in the next step (step S314). Exposure compensation methods are described, for example, in M. Uyttendaele, A. Eden and R. Skeliski (2001) "Eliminating ghosting and exposure artifacts in image mosaics," Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , p. II (https://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Uyttendaele01.pdf) can be used.

All images in the tile image set are moved and arranged according to the global transformation matrix calculated in step S302 (step S316). At this time, all images are joined using the seam information calculated in step S306. Further, a blending process may be performed at the time of joining. Various known techniques such as linear blending and multiband blending may be used for the blending process. For example, Richard Szeliski (2007), "Image Alignment and Stitching: A Tutorial", Foundations and Trends(R) in Computer Graphics and Vision: Vol. 2: No. 1, pp 1-104. (https://www. nowpublishers.com/article/Details/CGV-009) may be used.

The image (tile) synthesized in step S316 is saved as a temporary file in a storage area such as the HDD 32 of the computer device 30 (step S318). The above processing of steps S310 to S318 is executed for all tiles.

The temporary file saved in step S318 for each tile is read from a storage area such as the HDD 32 of the computer device 30, and sequentially written into the final file (step S320). As a result, the final virtual slide image that has been subjected to the overall composition processing is stored in a storage area such as the HDD 32 of the computer device 30.

FIGS. 12A and 12B are diagrams illustrating the effect of additional matching processing of region overlapping images in sequential joining processing. In the graph of FIG. 12A, the horizontal axis represents the registration order of images, and the vertical axis represents the number of images for which matching was attempted. In the graph of FIG. 12B, the horizontal axis represents the registration order of images, and the vertical axis represents the time required for matching. Here, three types of methods were used in the sequential bonding process. The first method is a method in which when a new image (New (N+1) image) is detected, matching is attempted with respect to all images that have been joined up to that point. The second method is a method that performs additional matching processing for region-overlapping images. The third method is a method in which when a new image (New (N+1) image) is detected, matching is attempted only with respect to the immediately previous spliced image (Last (N) image). In the first method, the number of images to be registered increases linearly as the images are registered later in the order. For this reason, the time required for matching is also increasing, and the user's waiting time gradually increases. In the second method, the number of images for which matching is attempted varies, but remains almost constant regardless of the order in which the images are registered. Therefore, the time required for matching is also almost constant. In the third method, the number of images for which matching is attempted is always 1, and therefore the time required for matching is also constant. However, when using the third method, the positional relationship between the new image (New (N+1) image) and images other than the previous merged image (Last (N) image) is not known, so Misalignment caused by repetition or optical system distortion cannot be corrected by Max Spanning Tree calculations or bundle adjustment. In this way, by using the second method (additional matching processing of region-overlapping images), it is possible to correct the deviation between images while keeping the time required for matching (user waiting time) constant.

FIGS. 13A and 13B are diagrams illustrating the effect of reducing misalignment between images by calculating Max Spanning Tree and simple bundle adjustment in sequential splicing processing. FIG. 13A shows a graph for pancreatic cancer HE-stained slides (320 images taken), and FIG. 13B shows a graph for liver HE-stained slides (59 images taken). These graphs show the relationship between the images at the end of the sequential splicing process, when Max Spanning Tree calculation and simple bundle adjustment were performed every 10 times, and when these were not performed. It shows the size of the deviation (RMS: root mean square). In both graphs, it can be seen that the deviation between images is significantly reduced when Max Spanning Tree calculation and simple bundle adjustment are performed. If the deviation is too large, it will be difficult for the user to confirm whether an appropriate spliced image has been generated, so it is effective to reduce the deviation by calculating the Max Spanning Tree or making simple bundle adjustments as necessary. It is.

FIGS. 14A and 14B are diagrams illustrating the effect of distortion correction in overall configuration processing. These particularly relate to the process of step S304 described above. FIG. 14A shows a graph for pancreatic cancer HE-stained slides (320 images taken), and FIG. 14B shows a graph for liver HE-stained slides (59 images taken). In addition, these figures show the magnitude of the deviation (RMS) between the case where distortion correction is performed by including a common distortion parameter in the parameter x of equation (1) mentioned above, and the case where distortion correction is not performed. It shows. Note that here, RMS is a numerical value indicating the deviation per pair of feature points in pixel units. It can be seen that when distortion correction is performed, the deviation is reduced to 0.5 pixel or less. This is a level of deviation that is invisible to the naked eye, indicating that the effect of distortion correction is extremely large.

(Configuration of microscope image information processing system)
As shown in FIGS. 1 and 2, a microscope image information processing system 1 that executes the microscope image information processing method according to the present embodiment may include a microscope 10, a camera 20, and a computer device 30. . As the microscope 10, camera 20, and computer device 30, existing general computer devices can be used. FIG. 15 is a diagram showing an example of the hardware configuration of the computer device 30. The computer device 30 includes a processor 31, an HDD 32, a RAM (Random Access Memory) 33, a ROM (Read Only Memory) 34, a removable memory 35 such as a CD, DVD, USB memory, memory stick, or SD card, and an input/output user interface (keyboard). , a mouse, a touch panel, a speaker, a microphone, an LED (light-emitting diode), etc.) 36, a wired/wireless communication interface 37 that can communicate with other computer devices, a display 38, and other hardware similar to general computer devices. ware configuration. The computer device 30 reads computer programs (camera software 40, joining software 42, and various other computer programs) stored in the HDD 32 and various data to be processed into a memory such as a RAM 33 and executes them. Each process of the microscope image information processing method according to the present embodiment described above can be realized.

Although the computer device 30 is illustrated as one device in FIGS. 1 and 2, it may be configured by two or more computer devices. For example, a first computer device stores images of each part of a slide glass specimen in the HDD 32, and a second computer device separate from this computer device uses a plurality of captured images stored in the first computer device. The process after the joining process may be executed sequentially. At this time, for example, the photographed image of the slide glass specimen temporarily stored in the HDD 32 of the first computer device is automatically transmitted to the second computer device by wired or wireless communication, and may be configured to use this as a trigger to execute subsequent sequential joining processing and overall configuration processing. Alternatively, for example, a first computer device executes the sequential joining process, and sends the sequentially joined images for which the sequential joining process has been completed and the data necessary for the overall configuration processing to the second computer device by wire or wirelessly. The information may be transmitted by communication. The second computer device may then execute the overall configuration process using the sent joined images and data necessary for the overall configuration process.

Furthermore, in this embodiment, the photographed images and joined images of each part of the slide glass specimen are described as being stored in the HDD 32, but they may be stored in other recording media.

According to the microscope image information processing method according to the present embodiment, the bonding process is performed sequentially while images of each part of the slide glass specimen are photographed, and images in the middle of bonding are sequentially displayed as previews to the user. The user can proceed with photographing the slide glass specimen while viewing the preview display and confirming whether the bonding process has been properly executed each time. This makes it possible to avoid the need for re-imaging the slide glass specimen due to the eventual failure of joining as in conventional off-line stitching.

Furthermore, according to the microscope image information processing method according to the present embodiment, unlike conventional real-time stitching, an automatic motorized stage or a dedicated camera is not required, and an existing imaging system (microscope, camera, computer device) can be used. It is possible to implement using As a result, the installation cost of the system is low, and virtual slides can be generated using familiar photographic images using a microscope that is familiar to daily use. Furthermore, as a conventional method for creating virtual slides, there is a device called a slide scanner that automatically scans slide glass specimens, but such devices are very expensive, but the microscope according to this embodiment According to the image information processing method, such a dedicated device is not required.

Furthermore, in order to photograph a fluorescent sample, an exposure time of about 500 m to 1 second is required. Also, if you want to generate virtual slides of multiply stained samples, manual switching of filters is required. For example, in conventional real-time stitching, it is difficult to change the shooting conditions on the fly. On the other hand, the microscope image information processing method according to the present embodiment can also be applied to fluorescent samples because the sample images are photographed one by one.

Furthermore, according to the microscope image information processing method according to the present embodiment, a preview image is generated as a reduced image, and a spliced image of the original size can be generated in the overall configuration processing, so that the amount of memory used can be reduced. Contributes to improving processing speed.

Although one embodiment of the present invention has been described so far, it goes without saying that the present invention is not limited to the above-described embodiment and may be implemented in various different forms within the scope of its technical idea.

Furthermore, the scope of the present invention is not limited to the exemplary embodiments shown and described, but also includes all embodiments that provide equivalent effects to the object of the present invention. Furthermore, the scope of the invention is not limited to the combinations of inventive features delineated by each claim, but may be defined by any desired combination of specific features of each and every disclosed feature. .

Note that the following configurations also belong to the technical scope of the present invention.
(1)
A method for processing microscopic image information performed by a computer system, the method comprising:
a first step of acquiring a captured image of a portion of the sample observed using a microscope and storing it in a storage area;
a second step of calculating feature point information that is information regarding feature points of the captured image when it is detected that the captured image is saved in the storage area;
Using the feature point information of the previous captured image and the feature point information of the captured image, a matching process is performed between the feature points of the previously captured image and the feature points of the captured image. The third step and
a fourth step of performing a joining process to generate a joined image based on the result of the matching process and the plurality of captured images that have been stored in the storage area to date;
a fifth step of displaying and outputting the spliced image;
including;
A microscope image information processing method, wherein the first step to the fifth step are repeated until a portion of the sample is photographed.
(2)
The microscopic image information processing method according to (1) above, wherein the spliced image generated in the fourth step is an image generated using the photographed image reduced at a predetermined ratio.
(3)
In the third step, a global transformation matrix of the photographed image stored in the first step is calculated based on a specific image among the photographed images stored in the storage area. , the microscope image information processing method according to (1) or (2) above.
(4)
The microscope image information processing method according to (3) above, wherein the global transformation matrix is calculated by Max Spanning Tree.
(5)
Microscope image information processing according to (4) above, wherein the global transformation matrix is adjusted by minimizing the reprojection error between all the feature points subjected to the matching process in the third step using the least squares method. Method.
(6)
For a region including the joined image generated by repeating the first step to the fifth step until the photographing of a portion of the sample is completed, the region and all the photographed images, The method for processing microscopic image information according to (1) above, further comprising the step of arranging the images according to the global transformation matrix after the fifth step.
(7)
a sixth step of dividing a region including the merged image generated by repeating the first step to the fifth step into a plurality of small regions until the photographing of a portion of the sample is completed; ,
For each of the plurality of small areas,
identifying one or more of the captured images that intersect with the small area to be processed;
a seventh step of arranging the small area to be processed and the identified one or more captured images according to a global transformation matrix;
The microscopic image information processing method according to any of (1) to (4) above, further comprising:
(8)
In the sixth step, bundle adjustment is performed on the spliced image generated by repeating the first step to the fifth step until imaging of a portion of the sample is completed. The microscope image information processing method according to (7) above, wherein the merged image is divided into a plurality of small regions.
(9)
The microscope image information processing method according to (8) above, wherein the bundle adjustment is performed by evaluating an equation defined by the Levenberg-Marquardt method below.

however,
x _i : Component of the transformation matrix of the image at processing time i J _i : Jacobian at processing time i (T on the right shoulder indicates transposition)
r _i : Error λ between feature points at processing time i: Real number greater than or equal to zero that is adjusted according to the size of the error I: Identity matrix (10)
The microscope image information processing method according to (8) above, wherein the bundle adjustment is performed by evaluating an equation defined by the Levenberg-Marquardt method below.

however,
x _i : Components of the image transformation matrix at processing time i and distortion correction parameters J _i : Jacobian at processing time i (T on the right shoulder indicates transposition)
r _i : Error λ between feature points at processing time i: Real number greater than or equal to zero that is adjusted according to the size of the error I: Identity matrix (11)
The microscopic image information processing method of (1) to (7) above, wherein the sample is a fluorescent sample.
(12)
The computer system includes a first computer device and a second computer device that is separate from the first computer device,
the first step is performed on the first computer device;
The microscopic image information processing method according to (1) to (8) above, wherein the second step to the fifth step are executed in the second computer device.
(13)
Performing a first step of acquiring a captured image of a portion of the sample observed using a microscope and storing it in a storage area,
When it is detected that the photographed image is stored in the storage area, a second step of calculating feature point information, which is information regarding the feature points of the photographed image, is executed;
Using the feature point information of the previous captured image and the feature point information of the captured image, a matching process is performed between the feature points of the previously captured image and the feature points of the captured image. Execute the third step,
a fourth step of performing a joining process to generate a joined image based on the result of the matching process and the plurality of captured images that have been stored in the storage area to date;
performing a fifth step of displaying and outputting the spliced image;
A microscope image information processing system that repeats the first step to the fifth step until a portion of the sample is photographed.
(14)
A computer program that causes a computer system to execute the microscope image information processing method of (1) to (12) above.

1...Microscope image information processing system 10...Microscope 20...Camera 30...Computer device 31...Processor 32...Hard disk device (HDD)
33...RAM
34...ROM
35... Removable memory 36... Input/output user interface 37... Communication interface 38... Display 40... Camera software 42... Bonding software 50, 50'... Captured image of slide glass specimen

Claims

A method for processing microscopic image information performed by a computer system, the method comprising:
a first step of acquiring a captured image of a portion of the sample observed using a microscope and storing it in a storage area;
a second step of calculating feature point information that is information regarding feature points of the captured image when it is detected that the captured image is saved in the storage area;
Using the feature point information of the previous captured image and the feature point information of the captured image, a matching process is performed between the feature points of the previously captured image and the feature points of the captured image. The third step and
a fourth step of performing a joining process to generate a joined image based on the result of the matching process and the plurality of captured images that have been stored in the storage area to date;
a fifth step of displaying and outputting the spliced image;
including;
A microscope image information processing method, wherein the first step to the fifth step are repeated until a portion of the sample is photographed.
2. The microscope image information processing method according to claim 1, wherein the merged image generated in the fourth step is an image generated using the captured image reduced at a predetermined ratio.
In the third step, a global transformation matrix of the photographed image stored in the first step is calculated based on a specific image among the photographed images stored in the storage area. The microscope image information processing method according to claim 1.
The microscope image information processing method according to claim 3, wherein the global transformation matrix is calculated by Max Spanning Tree.
5. Microscope image information according to claim 4, wherein the global transformation matrix is adjusted by minimizing reprojection errors between all feature points subjected to the matching process in the third step using a least squares method. Processing method.
For a region including the joined image generated by repeating the first step to the fifth step until the photographing of a portion of the sample is completed, the region and all the photographed images, 2. The microscope image information processing method according to claim 1, further comprising the step of arranging the following in accordance with a global transformation matrix after the fifth step.
a sixth step of dividing a region including the merged image generated by repeating the first step to the fifth step into a plurality of small regions until the photographing of a portion of the sample is completed; ,
For each of the plurality of small areas,
identifying one or more of the captured images that intersect with the small area to be processed;
a seventh step of arranging the small area to be processed and the identified one or more captured images according to a global transformation matrix;
The microscope image information processing method according to claim 1, further comprising:
In the sixth step, bundle adjustment is performed on the spliced image generated by repeating the first step to the fifth step until imaging of a portion of the sample is completed. 8. The microscope image information processing method according to claim 7, wherein the merged image is divided into a plurality of small regions.
9. The microscope image information processing method according to claim 8, wherein the bundle adjustment is performed by evaluating an expression defined by the following Levenberg-Marquardt method.

however,
x i : Component of the transformation matrix of the image at processing time i J i : Jacobian at processing time i (T on the right shoulder indicates transposition)
r i : Error λ between feature points at processing time i: Real number greater than or equal to zero that is adjusted according to the size of the error I: Identity matrix
9. The microscope image information processing method according to claim 8, wherein the bundle adjustment is performed by evaluating an expression defined by the following Levenberg-Marquardt method.

however,
x i : Components of the image transformation matrix at processing time i and distortion correction parameters J i : Jacobian at processing time i (T on the right shoulder indicates transposition)
r i : Error λ between feature points at processing time i: Real number greater than or equal to zero that is adjusted according to the size of the error I: Identity matrix
The microscope image information processing method according to claim 1, wherein the sample is a fluorescent sample.
The computer system includes a first computer device and a second computer device that is separate from the first computer device,
the first step is performed on the first computer device;
The microscope image information processing method according to claim 1, wherein the second step to the fifth step are executed in the second computer device.
Performing a first step of acquiring a captured image of a portion of the sample observed using a microscope and storing it in a storage area,
When it is detected that the photographed image is stored in the storage area, a second step of calculating feature point information, which is information regarding the feature points of the photographed image, is executed;
Using the feature point information of the previous captured image and the feature point information of the captured image, a matching process is performed between the feature points of the previously captured image and the feature points of the captured image. Execute the third step,
a fourth step of performing a joining process to generate a joined image based on the result of the matching process and the plurality of captured images that have been stored in the storage area to date;
performing a fifth step of displaying and outputting the spliced image;
A microscope image information processing system that repeats the first step to the fifth step until a portion of the sample is photographed.
A computer program that causes a computer system to execute the microscope image information processing method according to any one of claims 1 to 12.