JP2004253261A - Image collection/analysis system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically acquire numerous images from testpieces with mutually different fields of view, and in parallel with acquisition of the images, the acquired images are subjected to a prescribed image processing to distinguish whether or not there is a targeted structure in the images. <P>SOLUTION: A control means 4 acquires the images within a specified test range of testpieces (S3) by controlling (S2) the testpiece stage 2 of TEM1 when parameters are inputted (S1). Then, the acquired images are subjected to analytical processing immediately by the control means 4 (S6). In this way, acquisition of the images and analysis are simultaneously made in parallel. As for the analytical processing, as an index to express the probability that the targeted structure is reflected in the images, a processing which attaches brightness score based on the brightness of the images, a treatment which attaches profile score based on profile of figure in the images, a processing which attaches edge score based on linear figure in the images, and a processing which attaches comprehensive score based on these brightness score, profile score, and edge score are carried out. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image collection / analysis system that automatically and concurrently collects a sample image using an electron microscope and analyzes the collected image.
[0002]
[Prior art]
2. Description of the Related Art It is widely practiced to acquire enlarged images of various positions of a sample using a transmission electron microscope (TEM) and analyze the acquired images. For example, Japanese Patent Application Laid-Open No. 1-97359 discloses that an image of a certain field of view of a sample is acquired at a predetermined magnification, and then the sample is moved so that an image of a field of view adjacent to the field of view of the image acquired immediately before is obtained. It is disclosed that the operation of acquiring at a magnification of is repeated to continuously and successively acquire images in various fields of view of the sample. Also, analysis is performed by performing appropriate image processing on the acquired images.
[0003]
[Patent Document 1] Japanese Patent Laid-Open No. 1-97359
[0004]
[Problems to be solved by the invention]
By the way, when a large number of sample images with different fields of view are collected and analyzed, conventionally, the analysis of the sample images is usually performed after all the sample images have been collected. It took a very long time from the start of the analysis to the end of the analysis of all the sample images, which was not desirable.
[0005]
For example, if the size of a sample is 2 mm in diameter and an attempt is made to acquire images over the entire surface of the sample without any gaps, the number of images becomes enormous, and it takes a long time to acquire images. This is remarkable as the magnification of the TEM is increased, and is remarkable as the number of samples is large. In addition, in the related art, in addition to the above, it takes a long time to analyze the acquired image, so that it took a very long time to complete the analysis.
[0006]
In the analysis of the acquired images, pattern recognition can be performed relatively easily if the sample has a fixed structure such as a semiconductor or metal, so that a specific structure is included in the sample image. Although it is relatively easy to perform the process of analyzing whether there is a sample, if the sample is a biological sample and you want to analyze whether or not the target structure exists in the sample image, The analysis is difficult and time-consuming because the target structure is indefinite. Examples of such an analysis include, for example, an analysis of whether or not a virus is present in a sample image, and an analysis of whether or not a liposome (lipid bilayer) is present in a sample image.
[0007]
Therefore, the present invention automatically obtains a large number of images from a sample in mutually different fields of view, and for the obtained images, at the same time as the image acquisition, does the image have a target structure in the images? It is an object of the present invention to provide an image collection / analysis system capable of performing a predetermined image processing for determining whether or not the image collection is performed.
[0008]
[Means for Solving the Problems]
To achieve the above object, an image collection / analysis system according to claim 1 includes a transmission electron microscope, a camera that captures an image of a sample enlarged and projected by the transmission electron microscope, control means, The control means is characterized by performing an operation of simultaneously acquiring an image of a sample taken by a camera and analyzing the acquired image in parallel.
The image acquisition / analysis system according to claim 2 is the image acquisition / analysis system according to claim 1, further comprising an input unit, wherein the control unit acquires the image of the sample by using the input unit to determine an inspection range of the sample. When the number of acquired images is set, the set number of images is acquired within the set inspection range.
In the image acquisition / analysis system according to the third aspect, in the image acquisition / analysis system according to the first aspect, in the image analysis processing, the control unit determines whether or not a target structure is included in the image. As indices to be expressed, a process of giving a luminance score based on the luminance of an image, a process of giving a shape score based on the shape of a figure in an image, a process of giving an edge score based on a linear figure in an image, and these luminances It is characterized in that a process of giving a total score based on a score, a shape score, and an edge score is performed.
An image acquisition / analysis system according to a fourth aspect of the present invention is the image acquisition / analysis system according to the third aspect, further comprising a storage unit, and the control unit sets the brightness score and the shape score as conditions for storing data by the input unit. If any one of the edge score and the total score and the score value are set, only the image data and the image of the image in which the set score is equal to or more than the set score value are set. The data is stored in a storage means.
An image acquisition / analysis system according to a fifth aspect is the image acquisition / analysis system according to any one of the first to fourth aspects, further comprising a monitor, and wherein the control means is configured to complete analysis of all the acquired images. Is characterized in that an analysis result is displayed on a monitor.
[0009]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the invention will be described with reference to the drawings.
FIG. 1 is a diagram showing an embodiment of an image acquisition / analysis system according to the present invention, wherein 1 is a TEM, 2 is a sample stage, 3 is a camera, 4 is control means, 5 is storage means, and 6 is Input means 7 indicates a monitor.
[0010]
Although the TEM 1 has a well-known configuration, at least the magnification thereof can be controlled by the control unit 4. Since such a thing is well known, detailed description is omitted. The sample stage 2 of the TEM 1 has a sample holder on which a plurality of samples can be placed. Further, the position of the sample stage 2 with respect to the electron beam, that is, the position of the sample of the sample holder and the position of the sample to be irradiated with the electron beam can be controlled by the control means 4. Since such a sample stage is well known, a detailed description of its configuration will be omitted.
[0011]
The camera 3 is for acquiring an enlarged image of the sample, and here a TV camera is used. Therefore, it will be referred to as a TV camera 3 below.
The storage unit 5 is for storing predetermined information such as acquired images and analysis results of those images, and is preferably a large-capacity storage device such as a hard disk device.
The input means 6 is composed of a well-known input device such as a keyboard and a mouse. Here, it is assumed that the mouse can be left-clicked and right-clicked.
The monitor 7 is configured by a display device capable of displaying images, such as a color CRT and a color liquid crystal display device.
The control means 4, the input means 6, and the monitor 7 constitute a graphical user interface (GUI), and have various menus as described later.
[0012]
The control means 4 is for controlling the operation of each part of the image acquisition / analysis system in a centralized manner, and controls a magnification of the TEM 1, a position of the sample stage 2, or a TEM control software module for controlling a power supply. (Hereinafter, a software module is simply referred to as a module), a video capture module for capturing image data from the TV camera 3, and a predetermined image process for the obtained image data for analysis and data storage. Image processing module, an inspection display module for displaying items set during image acquisition and image processing on the monitor 7, an analysis result display module for displaying analysis results on the monitor 7, and the like. Various modules are mounted.
[0013]
The control means 4 can be constituted by only one computer system. For example, two personal computers are used, one is used for video capture and control of the TEM 1, and the other is used. One can be used for other image processing and display of the result.
[0014]
Therefore, for example, if a module for performing various types of image analysis is prepared as an image processing module, it is possible to cope with various types of analysis by mounting a module according to the purpose of the analysis. Also, by preparing various types of display modes for the analysis result display module, it is possible to display the analysis result in a mode desired by the user.
As described above, in this image collection / analysis system, it is possible to perform various image processing and display various results by exchanging modules.
[0015]
Hereinafter, the operation will be described. Here, the sample is a biological sample, and a module for performing image processing for determining the possibility that liposomes are present in the acquired image is used as the image processing module. .
[0016]
Before describing the operation, first, the validity of performing the image processing for determining the possibility that the liposome is captured in the acquired image will be briefly described.
Membrane proteins present in the cell membrane of living organisms play various roles in the activities of living organisms, and determining the structure of membrane proteins has great significance in biology, medicine, and pharmacy. Since it is a single large molecule, it must first be crystallized for accurate determination of its structure.
[0017]
However, the sample preparation conditions that can be crystallized for each type of membrane protein differ, and in order to crystallize, a huge number of samples must be prepared under various conditions and the presence or absence of crystals in each sample must be confirmed. Must.
[0018]
As a method of determining whether or not there is a crystal of a membrane protein in a sample, a method of preparing a three-dimensional crystal of the membrane protein and determining the presence or absence of the crystal by X-rays is generally used. Not all protein structures are necessarily reproduced, and two-dimensional crystals must be prepared for more accurate investigations.
[0019]
However, the presence or absence of a two-dimensional crystal cannot be confirmed by X-rays. Therefore, when analyzing the structure of the membrane protein using a two-dimensional crystal, an image of the sample is acquired using a TEM, and it is determined whether or not liposomes are present in the image or whether or not liposomes are likely to be present. Do that. This is because, if the sample contains membrane protein crystals, they are always formed on liposomes, so if there are liposomes in the image, or if there is a high possibility that liposomes are present, the sample Is based on the possibility that crystals of membrane proteins may be formed.
[0020]
In other words, this image processing does not directly determine the presence or absence of the crystal of the membrane protein. However, the sample determined to have the liposome, or the sample in which the liposome is likely to exist, and the position in the sample are determined. If it can be obtained, the presence or absence of the crystal of the membrane protein can be determined by examining the place in more detail by an appropriate method, and the conditions for forming the crystal of the membrane protein can be determined. It is.
[0021]
Hereinafter, the operation will be described with reference to a flowchart showing the flow of the operation shown in FIG. As will be apparent from the description below, in FIG. 2, the two horizontal lines below step S1 and the two horizontal lines above step S8 are two-series processes between these horizontal lines, that is, This shows that the processes of steps S2 to S5 and the processes of steps S6 and S7 are performed simultaneously in parallel.
[0022]
First, the operator adjusts the gain and the focus of the TV camera 3, and inputs predetermined parameters required for the analysis via the GUI (step S1). The parameters to be input here include, for example, designation of a sample from which an image is to be acquired, setting of an acquisition range of an image of the sample, setting of a magnification, setting of the number of images to be acquired for each sample, setting of the number of images, and saving. There are data conditions and the like, which will be described in detail below. Adjustment of the gain and focus of the TV camera 3 is a matter that is usually performed when an image is acquired by the TV camera.
[0023]
FIG. 3 is a diagram showing an example of a window displayed when performing parameter input. In FIG. 3, the title bar and the like of the window are not shown. This is the same in the following.
The inspection sample column 10 is a column for designating a sample from which an image is to be obtained. When the samples are placed on all the sample holders of the sample stage 2 and the images of all the samples are obtained, “all” is set. An item may be selected. To acquire an image only placed on a specific holder among the sample holders on the sample stage 2, select the item of “holder specification” and change the number of the holder. input. In the figure, “Specify Holder” is selected and “2, 3” is input. This means that images are acquired only from the samples placed on the second and third holders of the sample holder. Is shown.
[0024]
The inspection range column 11 is a column for setting from which range of the sample an image is obtained. The item “All” is an item to be selected when obtaining an image over the entire surface of the sample. In the case of acquiring an image only from a desired range in the range, the item of "range designation" is selected, and the center coordinates (x, y) of the range and the radius from the center coordinates are input. In the figure, the radius is in units of mm. In the figure, the coordinates (0, 0) are set such that an image is acquired with a radius of 0.1 mm from the center. The items set in the inspection range column 11 are reflected on the circle indicated by 12 on the right side. The outer circle indicates the entire field of view of the sample, and the inner circle indicates the range set in the “range designation” item of the inspection range column 11.
[0025]
The column 13 of display items under inspection is a column for setting what items are displayed during the operation of acquiring images and during the analysis of the acquired images. Can select an item to be displayed during the examination.
The item of the current image number is an item for displaying the number of the currently acquired image of the sample for which the image is currently acquired.
The total number of images is an item for displaying the total number of images acquired for the sample.
The sample number is an item for displaying the number of the sample currently targeted for image capture, that is, the number of the sample holder on which the sample is placed.
The coordinate is an item for displaying the position of the currently acquired image in the sample, and the coordinate.
The stored score is an item for displaying the set score when the score is set as a condition for storing the acquired image data and data such as the analysis result. The score will be described later.
The live image is an item for displaying an image currently captured by the TV camera 3.
[0026]
The data storage condition column 14 indicates whether or not to store data such as the result of performing image processing, which will be described later, on the acquired image, the image, the analysis result, and the like. This is a column for setting whether such a condition is provided.
As described below, in the image processing, three points of a luminance score, a shape point, and an edge point are assigned to each acquired image, and further, a total sum is obtained by taking a linear sum of the three points. However, when setting to save the data when the total score is equal to or more than one, the user selects "total score" and inputs a score value which is the threshold value in the rectangular box on the right side. ing. The same applies to the items of “luminance score”, “shape score”, and “edge score”. The item “all” is an item for setting to save all data such as acquired images and their analysis results, regardless of the score. In the figure, “Total score” is selected and a score of “0.45” is input. In this case, the image data having a total score of 0.45 or more obtained as a result of the image analysis is displayed. It will be stored together with the analysis results.
[0027]
The magnification column 15 is a column for setting the magnification of the TEM 1, and the operator can input a desired magnification.
The column 16 for the number of acquired images is a column for setting the number of images to be acquired for each sample set in the column 10 for the inspection sample, and is set in the column 11 for the inspection range for each sample. To obtain images of all the areas in the range without any gaps, the "automatic" item may be selected. To obtain a desired number of images in the range set in the inspection range column 11, "number setting" ”, And numerically input the number of sheets to be obtained in the rectangular box below the item.
[0028]
The high voltage setting column 17 is a column for setting whether to stop the supply of the high voltage to the TEM 1 or to keep the supply of the high voltage to the TEM 1 when the image capturing process in step S3 of FIG. 2 is completed. , OFF, the supply of the high voltage to the TEM 1 is stopped, and when ON is selected, the supply of the high voltage is continued.
[0029]
The OK button 18 is a button for confirming the input parameters, the registration button 19 is a button for registering the input parameter values, and the reading button 20 is operated when using the parameter values registered so far. The cancel button 21 is a button for canceling the input parameter value.
[0030]
When these parameters are entered and confirmed with the OK button 18, each parameter value is reflected on the corresponding module. For example, in the parameter input window shown in FIG. 3, each parameter value set in the inspection sample column 10, the inspection range column 11, the magnification 15 column, the number of acquired images column 16, and the high pressure setting column 17 is The parameters set in the data storage condition column 14 are reflected in the TEM control module, and are reflected in the image processing module, and the display item under inspection column 13, the data storage condition column 14, and the number of acquired images are set. Each parameter value set in 16 is reflected on the display module under inspection.
[0031]
When the input of the predetermined parameters is completed and the OK button 18 in FIG. 3 is operated to determine each parameter value, the control means 4 activates all the modules. Then, the TEM control module controls the magnification of the TEM 1 to the set magnification and moves the sample by controlling the sample stage 2 (step S2). First, the first sample is placed on the optical axis of the electron beam. The image acquisition position of is moved. Then, the control means 4 takes in the image signal from the TV camera 3 by the video capture module, digitizes the image signal, and stores it in the storage means 5 as image data (step S3). When the image data is stored, data such as the sample number of the sample, the image number, the coordinates at which the image was obtained, the date and time when the image was obtained, and the like are stored. The image number may be, for example, a serial number from 1 for each sample.
[0032]
By the way, the sample holder is provided with a copper grid in which a rectangular opening is regularly opened in a lattice shape in a thin copper plate in order to mount a sample. The beam may illuminate the copper portion just between the openings in the copper grid, in which case the image will be black and a meaningless image. When a portion of copper other than the opening portion of the copper grid is referred to as a meaningless region, it is known that the meaningless region of one sample holder is 80% or more. In other words, in the copper grid, the area of the opening is only about 20% of the entire area of the copper grid.
[0033]
Therefore, although not shown in FIG. 2, immediately before step S2, the range set in the inspection range column 11 of the sample holder set in the inspection sample column 10 of the parameter input window shown in FIG. It is checked which region is a meaningless region or which region is a meaningless region, and the capture of the image in step S3 in FIG. 2 is performed on a nonsense region, that is, an opening portion of the copper grid. To do.
[0034]
Examining a meaningless region or a non-significant region in the sample holder can be performed automatically or manually by an operator.
As a method of automatically checking a meaningless region, since the openings of the copper grid are regularly arranged, for example, the image is controlled in the range set in the inspection range column 11 of the parameter input window shown in FIG. There is a method in which the image is taken into the means 4 and the pattern of the black portion in the image is recognized by pattern recognition. Thus, the control unit 4 can recognize a non-significant area in the set inspection range.
[0035]
In the case of manual operation, a method of designating a non-significant region of the sample holder is used. For example, an image of the range set in the inspection range column 11 of the parameter input window is captured and displayed on the monitor 7, and while the operator observes the screen, the operator operates the input unit 6 to input the copper grid of the sample holder. There is a method of designating the opening portion of the image. This allows the control unit 4 to recognize an area other than a non-significant area in the inspection range.
[0036]
When recognizing an insignificant area or an insignificant area with respect to the set sample holder, the control means 4 executes the processing after step S2 in FIG. 2. First, the control means 4 executes the processing on the optical axis of the electron beam. Move the first image acquisition position of the first sample. Here, the first sample may be the smallest one of the sample holder numbers set in the column 10 of the test sample.
[0037]
Then, the control unit 4 successively acquires images in a range set in the inspection range column 11 of the parameter input window and a non-significant region of the sample holder (step S3). Here, the movement of the sample in step S2 will be roughly described as follows.
[0038]
Now, it is assumed that automatic is selected in the column 16 of the number of acquired images in the parameter input window in FIG. 3, and one non-significant area in the range set in the inspection range column 11 of the parameter input window is shown in FIG. Assuming that the rectangle is a rectangle indicated by a thick solid line in FIG. 4A, the control unit 4 determines an image capturing position in the non-significant area. FIG. 4A shows an example thereof. In FIG. 4A, a thin solid line rectangle indicates a capture range of each image.
[0039]
The control means 4, specifically the TEM control module, recognizes the set inspection range and the non-significant area, and the size of the field of view at the set magnification is fixed. As shown in FIG. 4A, it is possible to determine an image capturing position such that the field of view is completely filled in the non-significant area as shown in FIG. Can also be obtained.
[0040]
Therefore, in the case shown in FIG. 4A, the control means 4 performs the image capturing in each image capturing position in FIG. 4A in a predetermined order while moving the sample. The movement is performed by controlling the sample stage 2 to actually move the sample, and by controlling the deflection coil of the TEM 1 without moving the sample stage 2 to set the irradiation position of the electron beam to the next image capture position. There is an operation to change to.
[0041]
That is, controlling the sample stage 2 to actually move the sample is slow and inaccurate, so that the irradiation position of the electron beam is changed by the deflection coil within a range in which the irradiation position of the electron beam can be moved by the deflection coil. When the image capturing is completed within one deflectable range of the electron beam by the deflecting coil, the sample stage 2 is controlled to actually move the sample to the next electron beam. The operation of moving to the deflectable range, changing the irradiation position of the electron beam by the deflecting coil sequentially in a predetermined order within the deflectable range, and taking in the image is repeated.
As described above, the operation of moving the sample in step S2 of FIG. 2 includes the above two operations of movement.
[0042]
This will be described below with reference to the case shown in FIG. Now, for example, it is assumed that the deflectable range of the electron beam by the deflecting coil in the non-significant area shown in FIG. 4A is as shown by a broken-line rectangle in FIG. 4B. FIG. 4B shows six possible deflection ranges of the electron beam. In this case, for example, the control means 4 first controls the sample stage 2 to move the position of the sample to the center of the deflectable range of the electron beam at the lower left of FIG. The image capturing is performed in a predetermined order by changing the irradiation position of the electron beam within the deflectable range. In the case of FIG. 4B, 34 images are captured within the deflectable range of the electron beam.
[0043]
Then, when the capture of the image within the deflectable range of the electron beam is completed, the control unit 4 controls the sample stage 2 to change the position of the sample to the next deflectable range of the electron beam, for example, the lower right position. By moving the electron beam to the center of the deflectable range of the electron beam and changing the irradiation position of the electron beam within the deflectable range of the electron beam, images are captured in a predetermined order.
[0044]
Then, when the capture of the image within the deflectable range of the electron beam is completed, the control unit 4 controls the sample stage 2 to change the position of the sample to the next deflectable range of the electron beam, for example, the left side of the middle stage. By moving the electron beam to the center position of the deflectable range of the electron beam and changing the irradiation position of the electron beam within the deflectable range of the electron beam, images are captured in a predetermined order.
By repeating the above operation, an image can be obtained without gaps in the non-significant area.
[0045]
Note that the order in which the images are acquired within the deflectable range of one electron beam can be arbitrarily determined. The case shown in FIG. 4B will be described. For example, an image is sequentially acquired from the lower left end of the deflectable range of the electron beam to the right side. From the left end to the right.
[0046]
The above is the description of the operation of moving the sample in step S2 in FIG.
In the above description, automatic is selected in the field 16 of the number of acquired images in the parameter input window. However, when the number of images is set in the field 16 of the number of acquired images in the parameter input window, By generating random numbers or the like, the coordinates of the set number are determined within the range set in the inspection range column 11, and the positions of the determined coordinates are sequentially moved, and the image is displayed at each coordinate position. You only need to get it. If the determined coordinates are a meaningless area, the process of determining the coordinates may be repeated. In addition, it is natural that there are two operations as described above in the operation of moving the coordinates.
[0047]
When the acquisition of the image of the set inspection range is completed for one sample, the control means 4 controls the sample stage 2 to move the sample of the next sample holder on the optical axis, and Is repeated (step S4). With the above operation, a predetermined number of images of the set inspection range can be acquired for the set sample.
[0048]
When all the images have been captured, the control means 4 controls the high voltage of the TEM 1 by the TEM control module (step S5). That is, in the parameter input window, when ON is set in the high voltage setting column 17 in the parameter input window, the control unit 4 keeps the high voltage as it is, and when OFF is set, stops the high voltage.
[0049]
When acquiring one image, the control means 4 performs predetermined image processing on the image by the image processing module, and performs analysis and storage processing (step S6). Therefore, the process of step S6 is performed simultaneously and in parallel with the operations of steps S2 to S5.
[0050]
The image analysis process is as follows.
In this embodiment, as described above, an image in which a liposome may be present is found, but various structures are shown in the image of the sample. An example of such an image is shown in FIG. In the figure, A is a liposome, B is a support membrane, C is a portion where the support membrane is broken and has holes, D is a copper grid, and E is a drug. And F indicates an aggregate. The copper grid is a structure for mounting the sample, the support film is a thin film made of an organic substance that is pasted on the copper grid when the sample is mounted on the copper grid, and the aggregate is a protein molecule that has been randomly disordered Which are all well-known.
[0051]
They try to determine whether or not liposomes are or are likely to be in such an image, but the liposomes are amorphous and do not have a fixed shape, and there is no fixed pattern Therefore, identification by pattern matching is impossible. However, the present inventors have found that the structure of liposome has the following characteristics in an image.
(B) Luminance: Although it is darker than the supporting film serving as the background, it is not as black as aggregates and copper grids.
(B) Shape: The shape is irregular, but round and not jagged. That is, there are few contour irregularities.
(C) Pattern: There is no fixed pattern, but usually there is a long streak-like pattern inside.
[0052]
Therefore, by utilizing the above characteristics, it is determined whether or not there is a liposome in such an image, or whether or not it is likely. For that purpose, the image processing module acquires The following processing is performed on the image.
(1) Black portions in the image are excluded from the analysis range.
(2) White portions are excluded from the remaining range.
(3) Calculate a luminance score based on the average luminance and the variance of the luminance in the remaining area.
(4) Calculate the shape score based on the shape of the outer shape of the closed area in the image.
(5) An edge score is calculated based on the pattern in the closed area.
(6) The total score is calculated by linearly combining the brightness score, shape score, and edge score.
(7) The information of the four scores obtained above is added to the image and stored.
[0053]
[Process to exclude black and white parts]
According to the research of the present inventor, when a luminance histogram is created for an image in which a liposome is captured or an image in which a liposome is likely to be captured, a peak having two peaks is almost always obtained as shown in FIG. I found out. In this case, the gradation of the image is 256 gradations from 0 to 255, and it is assumed that all black is gradation 0 and all white is gradation 255.
[0054]
Then, the gray scale indicated by A in FIG. 6, that is, the part where the gray scale is smaller than the gray scale where the number of pixels between two peaks is minimum is a black part such as an aggregate or a copper grid, and is a target of image analysis. It turned out that it may be excluded from. From this, it can be seen that it is sufficient to create a histogram of the luminance and exclude pixels having a gray scale at or below which the number of pixels between two peaks is minimum as a black portion.
[0055]
Next, a method for excluding a white portion in an image, that is, a support film serving as a background or a portion where the support film is broken from the analysis target is, for example, a method in which a certain gradation is set in advance as a threshold value, and the threshold value is determined. Although it is possible to make a pixel having the above gradation a white portion, according to research, the gradation shown by C in FIG. 6, that is, the position of the higher gradation peak differs depending on the content of the image. It has been found that it is not appropriate to fix the threshold value in advance.
[0056]
Therefore, as a result of various studies, the maximum gradation at which the number of pixels is 0 is defined as B (FIG. 6), and the gradation difference d between the gradation B and the gradation C of the higher gradation peak is calculated. Then, from the gradation C of the peak having the higher gradation, a gradation D whose gradation is lower by the gradation difference d is calculated, and a portion having a gradation higher than this gradation D is defined as a white portion. Turned out to be good.
[0057]
Therefore, the control unit 4 performs the above-described processing by using the image processing module, and excludes black and white portions of the image from the analysis target. Therefore, a gray area having a gradation between A and D in the histogram of FIG. 6 remains. The gray area obtained in this way is called a blob.
[0058]
[Process of calculating luminance score]
It is highly possible that the blob obtained as described above contains liposomes or aggregates having a light color, but it is also possible that some blobs are present on the support membrane. Therefore, based on the brightness of the blob and the dispersion of the brightness, the likelihood that the liposome is captured in the blob is scored. This is the luminance score.
[0059]
By the way, as described above, liposomes are generally whitish in color than aggregates and darker than the support membrane. According to the study of the present inventor, it has been found that the average luminance of the supporting film changes depending on the image, but the ratio of the average luminance of the liposome to the average luminance of the supporting film does not change much. Here, as the average luminance of the support film, the average luminance of the white part obtained by the above-described processing of excluding the white part may be used.
[0060]
As described above, the liposome does not have a fixed pattern, but usually has a long streak-like pattern inside, and thus has a variation in luminance. When the luminance distribution of the liposome portion, that is, a histogram is taken, FIG. As shown in FIG. 7 (a), while a broad peak appears, the histogram of the brightness of the aggregates having a lighter color and of some spots on the supporting film is shown in FIG. 7 (b). As a result of the study, it was found that the peak was as narrow as possible.
[0061]
From the above, it is possible to determine the likelihood that the liposome is captured by scoring based on the average brightness of the blob, and further, by scoring based on the dispersion of the blob brightness, the liposome It can be seen that it is possible to determine the certainty of the image.
[0062]
For scoring based on the average brightness of the blob, for example, the average brightness of the blob and the average brightness of the support film, that is, the average brightness of the white portion of the image is obtained, and the ratio of the two, the average brightness of the liposome and the average of the support film are obtained. The smaller the difference from the luminance ratio, the higher the score, and the larger the difference, the lower the score. Specifically, when the average brightness of the blob is B, the average brightness of the support membrane is H, and the ratio of the average brightness of the liposome to the average brightness of the support membrane is R, for example,
1 / | (B / H) -R |
, And that value can be used as a score based on the average luminance of the blob as it is. The value of R may be obtained by experimentally determining the ratio between the average luminance of the liposome and the average luminance of the support film, and registering that value in the image processing module in advance.
[0063]
The scoring based on the blob luminance variance can be achieved by creating a blob luminance histogram and obtaining a higher score as the width of the peak, for example, the half-width of the peak is wider. What kind of score is given to what kind of peak width is arbitrary.
[0064]
As described above, two scores based on the blob brightness are obtained, and the control unit 4 sets the sum of the two scores as the brightness score of the image. The higher the luminance score, the higher the possibility that the liposome is captured. In other words, the luminance score is one of the indices indicating the likelihood that the target liposome structure is shown in the image.
[0065]
[Process of shape score calculation]
As described above, the shape of the liposome is indefinite, but it is round and not jagged, and the contour is less uneven. Therefore, when there is a structure in which a closed region is formed in the blob, it is possible to determine the likelihood that the liposome is captured in the blob based on the degree of irregularity of the contour of the closed region. That is, by performing scoring according to the degree of unevenness of the contour of the closed region in the blob, it is possible to determine the likelihood that the liposome is captured, and this is the shape score.
[0066]
There are various methods for calculating the shape score, and an example will be described.
First, a process of extracting a closed region is performed on the image in the blob. This process may use a known method. Then, a circumscribed figure in which the extracted closed region has no concave portion and has a minimum perimeter is created. Such a circumscribed figure is known as a convex circumscribed figure. For example, if a closed region L indicated by a solid line in FIG. 8 is now extracted, both ends of the concave portion are connected by a straight line as indicated by a broken line in the drawing. As a result, a figure as if rubber bands were applied to the closed area L is obtained. This is the convex hull circumscribed figure L '. Since the process for obtaining the convex hull circumscribed figure is well known, detailed description is omitted.
[0067]
Then, the ratio of the circumference of the extracted closed region L to the circumference of the convex hull circumscribed figure L ′ is determined, and a score is given in accordance with the ratio to obtain a shape score. In this case, the closer the ratio is to 1, the higher the score. This is because the contour of the liposome has few irregularities, and the ratio of the circumference of the contour to the circumference of the convex circumscribed figure of the contour may be considered to be close to 1. At this time, it is possible to arbitrarily determine what kind of score is obtained when the ratio of the circumference of the closed region to the circumference of the convex hull circumscribed figure.
[0068]
As described above, a shape score based on the shape of the closed area of the blob is obtained. Therefore, the higher the shape score, the higher the possibility that the liposome is captured. In other words, the shape score is one of the indices indicating the likelihood that the target liposome structure is shown in the image.
[0069]
[Process of edge score calculation]
As described above, there is no fixed pattern inside the liposome, but usually there is a long streak-like pattern inside. In the present specification, a pattern having a length equal to or longer than a predetermined length among such long streak-like patterns, that is, linear figures is referred to as an edge. From this, if scoring is performed according to the length of the linear figure of the blob, it is possible to determine the likelihood that the liposome is captured by this score. This is the edge score.
[0070]
There are various methods for calculating the edge score, and an example will be described.
First, a process of extracting a closed region is performed on the image in the blob, and further, a process of extracting a linear figure is performed within the extracted closed region. Since the process of extracting a closed region and the process of extracting a linear figure are well known, detailed description will be omitted.
[0071]
By the way, in the study of the present inventor, the length of the extracted linear figure is various, and if it is a liposome, it is relatively long, but the linear figure based on the aggregate, the stain of the support membrane, noise, etc. is generally short. It is confirmed that it is bent. Therefore, only the extracted linear figure whose length is equal to or larger than the threshold value is set as an edge. This threshold may be determined based on experiments, and the value may be registered in advance in the image processing module and used.
[0072]
Then, the areas of all edges in the closed region are obtained. For the area of the edge, for each edge, the number of pixels constituting the edge may be obtained, and the sum of the numbers of pixels may be determined as the area of all edges. Then, the ratio of the edge area to the area of the extracted closed region is obtained, and scoring is performed so that the ratio is high and the score is high. At this time, what kind of score is given to what ratio can be arbitrarily determined.
[0073]
As described above, an edge score based on the edge within the closed area of the blob is obtained. Then, the higher the edge score, the higher the possibility that the liposome is captured. In other words, the edge score is one of the indices indicating the likelihood that the target liposome structure is shown in the image.
[0074]
[Total score calculation process]
As described above, the control unit 4 obtains three types of scores for the acquired image: a luminance score, a shape score, and an edge score. Then, a total score is obtained based on these three types of scores. This total score can be defined by a linear sum of three types of scores, and the brightness score is I, the shape score is S, the edge score is E, and α, β, γ are coefficients, for example,
A = α × I + β × S + γ × E (1)
To obtain the total score A. Here, the values of the coefficients α, β, and γ can be arbitrarily determined, and if there is a score to be regarded as important, the coefficient of the score may be set to a large value. The values of these coefficients α, β, and γ may be determined in advance and registered in the control unit 4, more specifically, in the image processing module. This total score is also one of the indices indicating the likelihood that the target liposome structure is shown in the image.
[0075]
[Process for saving data]
As described above, a luminance score, a shape score, an edge score, and a total score are given to each of the acquired images. When the control unit 4 performs these four types of scoring for one image, the control unit 4 determines whether to save the image based on the conditions set in the data storage condition column 14 of the parameter input window. It is determined whether or not the image is satisfied, and only the image satisfying the data storage condition is stored. Images that do not satisfy the data storage conditions are deleted from the storage unit 5.
[0076]
For example, as shown in FIG. 3, in the data storage condition column 14 of the window, when it is set that the total score is set to 0.45 or more, the total score of the image that has just been scored is set. If the value is less than 0.45 points, the data is not stored and the data relating to the image stored in the storage means 5 is deleted. If the value is 0.45 or more, the data is stored in the storage means 5 so far. Four types of score values are added to the stored data relating to the image and stored. At the time when the image is obtained, data such as image data, the sample number of the sample, the image number, the coordinates at which the image was obtained, the date and time at which the image was obtained, etc. are added. Scores will be added and saved.
[0077]
However, if all images are stored in the data storage condition column 14, all images are stored regardless of the above four types of scores. At this time, it is natural that the above four types of scores are also added.
[0078]
According to the above-described storage processing, only the image having a high possibility of liposome being stored is saved, and the image having a low possibility is deleted from the storage means 5, so that the storage means 5 is effectively used. be able to.
[0079]
[Processing of display during inspection]
The control means 4 executes the processing of step S6 in FIG. 2 described above for all the acquired images (steps S6 and S7). Here, the acquisition of the image and the image analysis are performed in parallel. The window that is displayed when the user is present will be described.
[0080]
When image acquisition and image analysis are performed simultaneously in parallel, the control means 4 causes the display module under inspection to display a window for the item selected in the display item under inspection column 13 of the parameter input. FIG. 9 shows an example of the window display. Each item shown in FIG. 9 is as described above. In FIG. 9, the coordinates are indicated by asterisks, but it goes without saying that numerical values are actually displayed. The storage score is a score set in the data storage condition column 14 of the parameter input window. When "all" is selected in the data storage condition column 14, "All" is displayed in the storage score item. The stop button at the lower right of FIG. 9 is a button for ending this window display.
[0081]
In FIG. 9, the circle on the left side indicates the visual field range set in the column 11 of the inspection range of the parameter input, and points are displayed at positions where image acquisition has been performed so far.
Further, when a live image is selected in the field 13 of the display item under inspection, the control means 4 displays a window separately from the window display shown in FIG. An image is displayed.
[0082]
[Result display processing]
Next, the control means 4 displays a list of analysis results by the analysis result display module (step S8). FIG. 10 shows an example of the list display window.
[0083]
The window shown in FIG. 10 is divided into four regions: an analysis number display column 30, a data display column 31, an image display column 32, and a visual field display column 33. FIG. 11 is a diagram showing an example of the analysis number display column 30. In FIG. 11, the analysis number of the analysis currently stored in the storage means 5 is displayed, and the sample number is displayed for the analysis completed this time. I have. FIG. 11 shows that two analyzes have been performed so far, and the present analysis is the third analysis, and in this analysis, two samples having sample numbers 2 and 3 have been analyzed.
[0084]
Here, one analysis refers to a series of operations from the parameter input in step S1 of FIG. 2 to the result display in step S8. Therefore, this is the third analysis, and if the setting is made as shown in FIG. 3 in the column 10 of the test sample in the parameter input window, the display is as shown in FIG.
[0085]
In FIG. 11, the item of each analysis number and the item of the sample number can be selected by left-clicking with the mouse of the input means 6, and when the list display window shown in FIG. In the case of FIG. 11, "sample number 2" is automatically selected. Therefore, in FIG. 11, when the analysis result of the analysis number 2 is to be displayed, if “analysis 2” is selected, the sample numbers analyzed in the analysis 2 are displayed in a tree shape under “analysis 2”. The sample with the smallest sample number is automatically selected.
[0086]
The data display column 31 displays a list of various data related to the image acquired for the sample number selected in the analysis number display column 30 and stored as a result of the analysis by the image processing module. FIG. 12 is a diagram showing an example of the data display column 31. In FIG. 12, the image number of the image stored as a result of the analysis, the date and time of acquisition of the image, the coordinates on the sample from which the image was acquired, and the brightness of the image The score, the shape score of the image, the edge score of the image, and the total score of the image are displayed. Thus, data relating to one image is displayed on one line. The data display column 31 can be scrolled up and down.
[0087]
In the image display column 32, images obtained for the sample number selected in the analysis number display column 30 and stored as a result of analysis by the image processing module are displayed as thumbnails. The image display field 32 is scrollable up, down, left, and right.
[0088]
In the visual field display column 33, the visual field of the sample number selected in the analysis number display column 30 is displayed as a circle, and the position where the image was obtained is indicated by a dot.
[0089]
As described above, by displaying the result, it is possible to know what kind of image is obtained from which position of which sample, and how much each score is.
[0090]
Each of the thumbnail images displayed in the image display section 32 can be selected by left-clicking the input unit 6 with the mouse. When a certain thumbnail image is selected by left-clicking the mouse, the control unit 4 sets the thumbnail image. In the visual field display field 33, a cross mark is displayed at the position where the image is obtained. Therefore, it is possible to confirm at which position in the sample the selected image is obtained.
[0091]
In the image display column 32, a certain thumbnail image is selected by left-clicking the mouse, and then right-clicking the mouse to display data relating to the image or perform a desired operation on the image. It has been made.
[0092]
An example is as follows.
When a certain thumbnail image in the image display column 32 is selected and the mouse is right-clicked, for example, a menu window as shown in FIG. 13 is displayed. When an original image is selected, the original image of the image is displayed in the window. Is displayed. When the reproduction of the visual field is selected, the control unit 4 recognizes the sample number from which the image has been obtained and the coordinates thereof, and controls the sample stage 2 to move the coordinates so that the coordinates are located on the optical axis of the TEM 1. The live image is a menu for displaying an image currently captured from the TV camera 3. When observing a live image, if the high pressure is stopped by the processing in step S5, it is natural that the high voltage needs to be supplied to the TEM1.
[0093]
When the property is selected in the menu window of FIG. 13, the data displayed in the data display column 31 such as the coordinates at which the image was acquired, the acquisition date and time, the magnification, and the above four types of scores are displayed.
[0094]
The FFT in the menu window in FIG. 13 is a menu for performing a fast Fourier transform on the image. This menu is prepared so that it can be used to determine whether or not crystals of the membrane protein are present in the acquired image. The presence or absence of the crystals of the membrane protein is determined by the power spectrum obtained by the FFT. Can be determined.
[0095]
However, when performing the FFT, the size of the region to be subjected to the FFT must be the number of pixels of a power of 2, so that the region to be subjected to the FFT needs to be set. Further, when the acquisition magnification of the image is low, even if crystals of the membrane protein are present in the image, a power spectrum indicating the presence of the crystals cannot be obtained. Therefore, when the acquisition magnification is low, for example, the field of view reproduction is selected, the magnification is set to a high magnification, a live image is captured, an area to be subjected to FFT in the live image is set, and the FFT is executed. What should I do?
When the result of step S8 is displayed as described above, the analysis is completed.
[0096]
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said Embodiment, A various deformation | transformation is possible. For example, in the above embodiment, a case was described in which the likelihood that a liposome was captured in an acquired image was analyzed. However, the present invention provides, for example, determination of whether a virus is present in a sample, or protein in a sample. Can be applied to the determination of whether or not the target structure is present in the biological sample, such as the determination of the presence or absence of the molecule.
In the above embodiment, the TV camera is used. However, a slow scan camera or the like may be used instead of the TV camera.
[0097]
As described above, according to this image collection / analysis system, the acquisition of an image and the analysis of the acquired image are performed simultaneously in parallel, so that the time of one analysis can be shortened compared to the related art. Can be.
[0098]
In addition, the luminance of the image, a luminance score is assigned according to the variance of the luminance, a shape score is assigned according to the shape of the structure in the image, and an edge score is assigned according to the pattern in the image. Since the total score is given as a linear sum of the scores, it can be reliably determined whether or not an amorphous structure such as a liposome is present in the acquired image.
[0099]
Still further, the image and the predetermined data relating to the image can be stored for all the acquired images, but it is also possible to store only the type of the score and those satisfying the value of the score. Therefore, the storage means can be used efficiently and effectively.
[0100]
Further, when performing analysis, the operator only needs to perform the operation of inputting parameters, and thereafter, the image acquisition, the analysis of the acquired image, and the display of the analysis result are automatically performed, so that the burden on the operator is reduced. You.
[Brief description of the drawings]
FIG. 1 is a diagram showing an embodiment of an image collection / analysis system according to the present invention.
FIG. 2 is a flowchart showing a flow of operation.
FIG. 3 is a diagram illustrating an example of a window displayed when performing parameter input in step S1 of FIG. 2;
FIG. 4 is a view for explaining a sample movement in step S2 of FIG. 2;
FIG. 5 is a diagram for explaining various structures appearing in an acquired image.
FIG. 6 is a diagram for explaining a process of excluding a black portion and a white portion from an image, which is performed in the image analysis process in step S6 of FIG. 2;
FIG. 7 is a diagram for explaining a process of calculating a luminance score performed in the image analysis process of step S6 in FIG. 2;
FIG. 8 is a diagram for explaining a process of calculating a shape score performed in the image analysis process in step S6 of FIG. 2;
FIG. 9 is a diagram illustrating an example of a window displayed when image acquisition and image analysis are performed simultaneously in parallel.
FIG. 10 is a diagram illustrating an example of a window that displays an analysis result.
11 is a diagram showing a display example of an analysis number display column 30 of the window shown in FIG.
12 is a diagram showing a display example of a data display column 31 of the window shown in FIG.
13 is a diagram showing an example of a menu window displayed when one thumbnail image is selected in the image display column 32 of the window shown in FIG.
[Explanation of symbols]
1 TEM, 2 sample stage, 3 TV camera, 4 control means, 5 storage means, 6 input means, 7 monitor.

Claims (5)

A transmission electron microscope,
A camera for capturing an image of the sample enlarged and projected by the transmission electron microscope,
Control means;
At least,
An image collection / analysis system, wherein the control unit performs an operation of executing, in parallel, acquisition of an image of the sample captured by the camera and analysis of the acquired image. 2. The image acquisition / analysis system according to claim 1, further comprising an input unit.
The control unit acquires the image of the sample, when the inspection range of the sample and the number of acquired images are set by the input unit, acquires the number of images set within the set inspection range. Featured image acquisition / analysis system The image collection / analysis system according to claim 1,
In the image analysis processing, the control means assigns a luminance score based on the luminance of the image as an index indicating the likelihood that the target structure appears in the image, based on the shape of the figure in the image Image collection by performing a process of assigning a shape score, a process of assigning an edge score based on a linear figure in an image, and a process of assigning a total score based on the luminance score, the shape score, and the edge score. / Analysis system. The image collection / analysis system according to claim 3,
Further comprising storage means,
The control unit may set any one of the luminance score, the shape score, the edge score, and the total score as a condition for storing the data by the input unit, and when the value of the score is set, the set score is used as the condition. An image collection / analysis system characterized in that image data and predetermined data relating to the image are stored in a storage means only for an image having a score value or more. The image acquisition / analysis system according to any one of claims 1 to 4,
It also has a monitor,
An image collection / analysis system, wherein the control means displays an analysis result on a monitor when analysis of all acquired images is completed.

Images