JP2010117175A - Image processing system, image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing system capable of detecting a target particle image from an image of a low S/N ratio in a short time with high precision. <P>SOLUTION: The image processing system 1 includes: a partial image extracting part 11 for extracting a plurality of partial images from the whole image; an image memory part 161 in which the image data of a reference particle image simulating the target particle image is stored; a peak detecting part 12 for detecting the peak of the similarity distribution between the respective partial images and the reference particle image with respect to the respective partial images; a vector calculating part 13 for calculating the position setting vector reaching the position of the peak from the predetermined reference positions in the respective partial images; and a peak distribution forming part 14 for setting the position of the peak at the position indicated by the position setting vector to form a two-dimensional peak distribution. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing technique capable of detecting a large number of particle images appearing in an image, and in particular, an image processing technique capable of detecting a particle image appearing in an image having a low S / N ratio such as an electron microscope image. About.

  As a conventional technique for analyzing the three-dimensional structure of biopolymers such as protein complexes, nucleic acids, lipids or polysaccharides, a sample prepared by crystallizing biopolymers is prepared, and this sample is analyzed by X-ray crystal structure analysis. Technology is used. However, since this technique requires crystallization of a sample, this technique could not be applied to analysis of a sample that is difficult to crystallize such as membrane protein. In addition, although the nuclear magnetic resonance (NMR) method does not require crystallization of a sample, it is generally difficult to analyze a biopolymer having a large molecular weight.

  Thus, as a method that does not require crystallization of a sample, a single particle structure analysis method has been proposed in which protein structure analysis is performed using a cryogenic transmission electron microscope (Cryo-TEM). This single particle structure analysis method uses a high-resolution TEM image obtained from a sample cooled to a cryogenic temperature in a cryogenic transmission electron microscope. The sample contains a large number of particulate biopolymers (hereinafter referred to as “particles”), and these biopolymers are oriented in random directions. The single particle structure analysis method is a method of cutting out a local image (hereinafter referred to as “particle image”) representing individual particles from the TEM image and estimating the three-dimensional structure of the particles based on these particle images. According to this method, the noise component of the image can be reduced by averaging the particle images of particles directed in the same direction.

For example, Non-Patent Document 1 (Sato Main Tax et al .: “Protein structure analysis without using a single particle analysis method: taking the structure of a voltage-dependent Na ⁺ channel as an example” as a prior art document relating to a single particle structure analysis method, Electron microscope, 37 (1), 40-44, 2002).

  Hereinafter, an example of the single particle analysis method will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline of the single particle analysis method described in Non-Patent Document 1. The electron microscopic image of the biopolymer is an image containing a lot of noise components with limited resolution in order to prevent damage to the sample due to electron beam irradiation. When a biopolymer that is easily damaged by electron beams such as protein is imaged, the S / N ratio of the electron microscope image is low, so that it is difficult to fully exhibit the original resolution of the electron microscope. Therefore, there is a problem that not only sufficient information cannot be obtained from one electron microscope image, but also the three-dimensional structure of the biopolymer is not easily reconstructed. Single particle analysis is attracting attention as a method for overcoming such problems.

  In the single particle analysis method, first, a large number of particles appearing in an electron microscope image (ELECTRON MICROGRAPH) of a sample are specified. The particles are identified by the naked eye. Next, a local image (particle image) of each of these particles is cut out from the electron microscope image. Next, alignment and rotation alignment are executed for these local images. Thereafter, the particle images are classified into a plurality of groups according to the direction in which the particles are directed. A two-dimensional averaged image (projected image) with a high S / N ratio is obtained by averaging the particle images in the same group, and the three-dimensional structure of the particles is estimated using the two-dimensional averaged image. The unstained electron microscope image by the freezing method is thought to give a projected image reflecting the intramolecular density. Therefore, an image (sinogram) projected in one dimension from a two-dimensional averaged image with a high S / N ratio is generated, and these sinograms are compared with each other to compare the three-dimensional particles appearing in each two-dimensional averaged image. The upper azimuth angle (Euler angle) and the shared rotation axis can be estimated (lower left in FIG. 1). Then, the three-dimensional structure of the particles is reconstructed based on the two-dimensional averaged image, the azimuth angle, and the shared rotation axis. A method for reconstructing a three-dimensional structure is disclosed, for example, in Non-Patent Document 2 (Joachim Frank, “Three-Dimensional Electron Microscopy of Macromolecular Assemblies”, Oxford University Press, 2006, pp. 193-276).

  Furthermore, a reprojection image is generated from the three-dimensional structure obtained in this way, and after aligning the two-dimensional averaged image using the reprojection image, a new two-dimensional averaged image is obtained by further averaging. Generate. Then, the three-dimensional structure of the particles is reconstructed based on the new two-dimensional averaged image. It is possible to estimate the three-dimensional structure with high resolution by repeatedly executing the process of positioning using such a reprojection image and the reconstruction of the three-dimensional structure.

In addition to Non-Patent Document 1 and Non-Patent Document 2, Patent Document 1 (Japanese Patent Laid-Open No. 2008-134212) is cited as a prior art document relating to a single particle analysis method.
Sato Main Tax, Yutaka Ueno, Toshihiko Ogura, Yoshinori Fujiyoshi: “Protein structure analysis without using crystals by single particle analysis: an example of the structure of voltage-dependent Na + channel”, Electron Microscope, Vol. 37, No. 1, 40-44 (2002). Joachim Frank, "Three-Dimensional Electron Microscopy of Macromolecular Assemblies", Oxford University Press, pp. 193-276 (2006). van Heel M., "Multivariate statistical classification of noisy images (randomly oriented biological macromolecules)", Ultramicroscopy, vol.13, pp.165-183 (1984). van Heel M., "Classification of very large electron microscopical image data sets", Optik, vol. 82, pp. 114-126 (1989). JP 2008-134212 A

  As described above, in the single particle analysis method, in order to cut out a particle image from an electron microscope image, it is necessary to identify a large number of particles appearing in the electron microscope image, but each particle is identified by the naked eye. . However, when the S / N ratio of the electron microscope image is low and the contrast (the magnitude of the luminance difference between the brightest part and the darkest part of the electron microscope image) is low, noise is detected from the electron microscope image. It is very difficult to identify thousands to tens of thousands of buried particles with the naked eye, and the identification requires a lot of time and skill.

  Instead of identifying the particles with the naked eye, it is possible to identify the particles in a short time by using image processing for searching for particles from an electron microscope image using a reference image (template) by a pattern matching method. However, the simple pattern matching method has a problem that the particle detection accuracy is very low.

  In view of the above, an object of the present invention is to provide an image processing system, an image processing method, a program, and a recording medium capable of detecting a particle image such as a biopolymer from an image having a low S / N ratio with high accuracy and in a short time. Is to provide.

  According to the present invention, a partial image extraction unit that extracts a plurality of partial images from an entire image including a plurality of target particle images, and an image storage unit that stores image data of reference particle images simulating the target particle images Calculating a similarity distribution between each partial image and the reference particle image for each partial image, and detecting a peak of the similarity distribution; and each partial image for each partial image A vector calculation unit for calculating a position setting vector from a predetermined reference position to the peak position, and setting the peak position at a position specified by the position setting vector for the region of the entire image An image processing system including a peak distribution generation unit that generates a two-dimensional peak distribution is provided. In this image processing system, the partial image extraction unit extracts the plurality of partial images from the whole image such that partial images adjacent in a predetermined direction among the plurality of partial images overlap each other.

  According to the present invention, (a) extracting a plurality of partial images from an entire image including a plurality of target particle images, and (b) storing image data of reference particle images simulating the target particle images. Reading from the unit, calculating a similarity distribution between each partial image and the reference particle image for each partial image, and detecting a peak of the similarity distribution; (c) Calculating a position setting vector from a predetermined reference position in each partial image to the position of the peak; and (d) the peak at a position designated by using the position setting vector for the region of the entire image. And a step of generating a two-dimensional peak distribution by setting the position of the image processing method. In this image processing method, a plurality of partial images are extracted from the entire image so that partial images adjacent in a predetermined direction among the plurality of partial images overlap.

  Further, according to the present invention, there is provided a program that is read from a computer-readable recording medium and causes the computer to execute image processing. The image processing includes partial image extraction processing for extracting a plurality of partial images from an entire image including a plurality of target particle images, and reading out image data of reference particle images simulating the target particle images from an image storage unit, Calculating a similarity distribution between each partial image and the reference particle image for the partial image, and detecting a peak of the similarity distribution; a predetermined value in each partial image for each partial image; A vector calculation process for calculating a position setting vector from a reference position to the peak position, and two-dimensionally setting the peak position at a position designated by using the position setting vector for the entire image area Peak distribution generation processing for generating a peak distribution, and in the partial image extraction processing, partial images adjacent in a predetermined direction among the plurality of partial images are overlapped with each other. Wherein from the whole image a plurality of partial images are extracted.

  Furthermore, according to the present invention, a recording medium on which the program is recorded is provided.

  All of the image processing system, the image processing method, and the program according to the present invention use a plurality of partial images extracted from an entire image including a plurality of target particle images. Since these partial images are extracted so that the partial images adjacent in the predetermined direction overlap, the adjacent partial images can include the same target particle image. In the image processing system, the image processing method, and the program according to the present invention, the peak of the similarity distribution between each partial image and the reference particle image is detected, and the position from the reference position to the peak position in each partial image A setting vector is calculated. Then, a two-dimensional peak distribution is generated by setting the peak position at the position specified by the position setting vector for the entire image area. Since the position setting vector is a vector indicating a region where the existence probability of the target particle image is relatively high, in the two-dimensional peak distribution, the peak is likely to be concentrated at the position of the target particle image. Therefore, even from an image with a low S / N ratio, it is possible to detect the target particle image with high accuracy and in a short time based on the two-dimensional peak distribution.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate.

(Configuration of image processing system)
FIG. 2 is a functional block diagram showing a schematic configuration of the particle structure analysis apparatus 1 which is an image processing system according to an embodiment of the present invention. The particle structure analysis apparatus 1 includes a particle image detection unit 2 and a three-dimensional structure analysis unit 3. FIG. 3 is a functional block diagram showing a schematic configuration of the particle image detection unit 2.

  As shown in FIG. 2, an entire image that is an electron microscope image is input to the particle image detection unit 2. This electron microscope image is a TEM (Transmission Electron Microscope) image obtained by imaging a sample containing a biopolymer such as a protein complex, nucleic acid, lipid, or polysaccharide with a transmission electron microscope. The particle image detection unit 2 specifies the position of the observed particle image (hereinafter referred to as “target particle image”) appearing in the TEM image with high accuracy and in a short time, and gives the result to the three-dimensional structure analysis unit 3. The three-dimensional structure analyzing unit 3 has a function of estimating the three-dimensional structure of the target particle image specified by the particle image detecting unit 2. The function of the three-dimensional structure analysis unit 3 is not particularly limited as long as it uses the technology disclosed in Non-Patent Document 1 and Patent Document 1, for example.

  The particle image detection unit 2 and the three-dimensional structure analysis unit 3 may be incorporated in a single device, or may be a small-scale or large-scale computer network (for example, a LAN (Local Area Network) or a WAN). (Wide Area Network)).

  As shown in FIG. 3, the particle image detection unit 2 includes an image input unit 10, a partial image extraction unit 11, a peak detection unit 12, a vector calculation unit 13, a peak distribution generation unit 14, a particle image search unit 15, and a data storage. A unit 16, a particle image extraction unit 17, an image classification unit 18, an averaging processing unit 19, an update processing unit 20, and an image memory 21. The controller 22 has a function of controlling the operations of the function blocks 10 to 21 to control the entire process. The data storage unit 16 includes an image storage unit 161, a reference image storage unit 162, a peak distribution image storage unit 163, and a derived image storage unit 164.

  All or part of the functional blocks 10-15, 17-20, and 22 of the particle image detection unit 2 in FIG. 3 may be realized by hardware such as a semiconductor integrated circuit, or a non-volatile memory or an optical disk. It may be realized by a program or a program code recorded on the recording medium. Such a program or program code causes a computer having a processor such as a CPU to execute all or part of the processing of the functional blocks 10 to 15, 17 to 20, and 22.

  The data storage unit 16 includes a recording medium (for example, a semiconductor memory or a magnetic recording medium) such as a volatile memory or a nonvolatile memory, and a circuit or a program for writing and reading data on the recording medium. Can be configured. The various storage units 161 to 164 may be configured in advance on a predetermined storage area of the recording medium, or may be configured on an appropriate storage area assigned when the particle image detection unit 2 operates.

  Referring to FIG. 3, an electron microscope image including a large number of target particle images is input to the image input unit 10 as an entire image. The image input unit 10 may be a receiver that receives a transfer signal representing an entire image, or may be a device that reads data of an entire image from a recording medium. The image input unit 10 stores the input whole image data in the image storage unit 161.

The partial image extraction unit 11 reads an entire image including a plurality of target particle images from the image storage unit 161 in response to an instruction from the controller 22, and the partial images P _{1 to} P _M (M Is a positive integer). As described later, the partial image _P 1 to P partial image _P i adjacent to each other in a predetermined direction among the _M, the partial image _P 1 to P _M as _{P i + 1} overlap with each other is extracted. As a result, the adjacent partial images P _i and P _{i + 1} can include all or part of the same target particle image.

The reference image storage unit 162 stores reference images R _{1 to} R _N (N is a positive integer) including reference particle images simulating target particle images that can be included in the entire image. In the present embodiment, these reference images R _{1 to} R _N are images that represent the same type of particle images in which the directions of the particles are different. For example, the reference image R ₁ can be an image simulating the front of the target particle, the reference image R ₂ can be an image simulating the bottom surface of the target particle, and the reference image R ₃ can be an image simulating the side surface of the particle.

For each partial image P _j , the peak detection unit 12 calculates a two-dimensional distribution (hereinafter referred to as “similarity distribution”) of the similarity between the partial image P _j and the reference particle image, and the similarity is calculated. Detect peaks that appear in the distribution.

More specifically, first, the peak detecting section 12 reads the reference image _R 1 to R _N from the reference image storage unit 162. As described above, each of the reference images R _{1 to} R _N includes a reference particle image that imitates a target particle image. Next, the peak detection unit 12 while moving the respective reference image R _k for each partial image P _j, by sequentially calculating the similarity between each reference image R _k and the partial images P _j A similarity distribution DS _{j, k} (x, y, θ) is obtained. Here, x, y is the amount of movement in the x and y directions of each reference image R _k against the partial images P _j (shift amount). Further, θ is a rotational movement amount around the point (x, y) of each reference image R _k with respect to each partial image P _j . Then, the peak detector 12 detects a peak appearing in the similarity distribution DS _{j, k} (x, y, θ).

For example, the peak detector 12, similar between with sequentially selecting a target pixel region out of the partial images P _j, for each said target pixel area, and the reference image R _k and an image region including the target pixel region The similarity distribution DS _{j, k} (x, y, θ) is obtained by calculating the degree. In this case, x and y are shift amounts from the reference position of each reference image R _k to the target pixel region with respect to each partial image P _j , and θ is a rotational movement amount around the target pixel region. The target pixel region may be sequentially selected in units of one pixel, or may be sequentially selected in units of a plurality of pixels such as 4 pixels × 4 pixels. The similarity is preferably a known cross-correlation coefficient, but is not limited to this. For example, an arithmetic function (for example, 1 / X ² : X is a residual sum of squares) having a residual sum of squares between an image region including a target pixel region and a reference particle image as a variable can be used as the similarity. .

For each partial image P _j , the peak detection unit 12 detects a peak having a similarity degree equal to or greater than a threshold from the similarity distribution DS _{j, k} (x, y), and generates a peak distribution generation using the detection result as a vector distribution unit 13 Part 14 is given. Here, only a single peak having the maximum similarity for each partial image P _j may be detected, or a plurality of peaks may be detected for each partial image P _j . Hereinafter, it detected only a single peak with a maximum similarity for each partial image P _j, a method for detecting a target particle image by using the peak is referred to as "MRP (Multi-Reference Pickup) method", each A method of detecting at least one peak in the partial image P _j and detecting the target particle image using the peak will be referred to as “MRMP (Multi-Reference Multiple Pickup) method”.

Vector calculation unit 13, a predetermined reference position in the local region of the partial images P _j for each partial image P _j (e.g., the center of the partial image P _j) to calculate the position setting vector extending from the position of the peak Pk . A set of each peak Pk and the corresponding position setting vector is given to the peak distribution generation unit 14.

Peak distribution generating unit 14 sets the position of the peak Pk at the position specified by the position setting vector for each partial image P _j, to generate a two-dimensional peak distribution corresponding to the entire image area. Here, the position of the peak Pk is set to a position specified by the position setting vector position of each partial image P _j in the entire image region (global position) as a reference point. The two-dimensional peak distribution is stored in the peak distribution image storage unit 163. More specifically, the peak distribution generation unit 14 uses a predetermined reference position (x _j , y _j ) of each partial image P _j in the entire image region as a position setting vector calculated for each partial image P _{j. v = (δx j, δy j} ) only shifted to shifted reference position _{P (x j + δx j,} y j + δy j) the positioning vector _{v = (δx j, δy j} ) and the position specified by .

  Then, the particle image search unit 15 reads the two-dimensional peak distribution from the peak distribution image storage unit 163, and the peak density (hereinafter referred to as “peak density”) appearing in the two-dimensional peak distribution is an area where the threshold value is greater than or equal to the threshold value. A region where the existence probability of the target particle image is high) is searched to detect the position of each target particle image. Here, the peak density means the number of peaks per unit area. The region where the peak density is greater than or equal to the threshold is a region where the two-dimensional peak distribution is localized. In the two-dimensional peak distribution generated based on the entire image with a low S / N ratio, it is expected that a large number of peaks appear in the same region as the target particle image buried in noise or in the peripheral region. The existence probability of the target particle image is high in the region where is high.

  Further, the particle image search unit 15 has a function of searching for a region where the similarity of the peak position appearing in the two-dimensional peak distribution is equal to or greater than a threshold, and detecting the position of the target particle image based on the search result. Here, when the positions of a plurality of peaks appearing in the two-dimensional peak distribution overlap, the particle image search unit 15 determines the target particle image based on the addition result (cumulative similarity) of the similarities of the plurality of peaks. Detect position. In an image region having a relatively high S / N ratio, a plurality of peaks may appear at the same position, and the probability that the target particle image exists at that position is very high. The particle image search unit 15 can reliably detect the position of such a target particle image.

  As shown in FIG. 3, the particle image search unit 15 includes a peak density calculation unit 150 and a position detection unit 151. The peak density calculation unit 150 has a function of counting the number of peaks in the search window while scanning the search window with respect to the two-dimensional peak distribution, and sequentially calculating the peak density based on the counted value. Yes. The position detector 151 (i) when the peak density is equal to or higher than the first threshold, or (ii) when the peak position similarity or cumulative similarity in the search window is equal to or higher than the second threshold. It is determined that the target particle image is located at the position of the search window. Alternatively, the position detection unit 151 can also determine that the target particle image is located at the position of the search window only when both detection conditions (i) and (ii) are satisfied. The user can adjust the detection accuracy by setting the first threshold value and the second threshold value to arbitrary values according to the image quality of the entire image. The particle image search unit 15 gives data indicating the position of the target particle image detected in this way to the particle image extraction unit 17.

  The search window has a circular shape or a rectangular shape including at least one pixel, but is not particularly limited.

  The particle image extraction unit 17 extracts (cuts out) a particle image including the target particle image at the position detected by the particle image search unit 15 from the entire image, and the data of these particle images is transmitted via the image memory 21 in FIG. Has a function of outputting to the three-dimensional structure analysis unit 3. The three-dimensional structure analysis unit 3 in FIG. 2 estimates the three-dimensional structure of the target particle image based on the particle image input from the particle image detection unit 2.

  The particle image detection unit 2 includes functional blocks 18 to 20 that improve the quality of the reference particle image stored in the reference image storage unit 162 based on the particle image extracted from the entire image.

The image classification unit 18 has a function of generating and classifying derived images from particle images. Under the control of the controller 22, the image classification unit 18 compares the particle image input from the particle image extraction unit 17 with the reference images R _{1 to} R _N in the reference image storage unit 162, thereby obtaining one or more derived images. Classify into groups.

Specifically, the image classification unit 18 peaks between the reference images R _{1 to} R _N in the reference image storage unit 162 based on each particle image S (i) input from the particle image extraction unit 17. Derived images S (i, 0) to S (i, P _i ) (P _i is a positive integer) having similarities are generated. Here, i is a number that uniquely identifies the particle image. Mathematically, the derived image mathematically indicates the relative position of each particle image S (i) with respect to the reference images R _{1 to} R _N , the X-axis translation direction (horizontal pixel direction), and the Y-axis translation direction (vertical pixel direction). And an image obtained by moving each in the angular direction. When a cross-correlation coefficient is used as the similarity, an inner product of a vector whose element is the pixel value of each reference image R _k and a vector whose element is the pixel value of each derived image, that is, an amount proportional to the correlation coefficient It can be calculated as a similarity. The generated derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) (T is a positive integer of 2 or more) are stored in the derived image storage. Stored temporarily in section 164.

Next, the image classification unit 18 reads the derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) from the derived image storage unit 164. These derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) are classified into one or more groups. Specifically, the image classification unit 18 generates vectors for each of the derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ). Process. The derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) are two-dimensional, each having a horizontal pixel number n and a vertical pixel number m. If it is an image, in the vectorization process, an n × m-dimensional position vector having n × m pixel values of each two-dimensional image as elements is generated.

Derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) are respectively represented in the n × m-dimensional vector space by corresponding position vectors. Positioned on. The image classification unit 18 converts the derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) into an n × m-dimensional vector space (pixels). It can be grouped according to the proximity in the space. For example, in this vector space, it is possible to extract a predetermined number or more of derived images that are distributed in a local space so that the mutual distance is equal to or less than a threshold value as one group.

  Alternatively, derived images that are distributed in a localized space in an image space conjugate with the pixel space can be extracted as one group. If the image space dimension is lower than the pixel space dimension, classify the derived image in the image space, and if the pixel space dimension is lower than the image space dimension, classify the derived image in the pixel space. Is preferred. Thereby, the classification process of the derived image can be performed efficiently. Such a classification method is known as a hierarchical ascendant classification using correspondence analysis. Non-patent literature 3 (van Heel M., “Multivariate statistical classification of noisy images (randomly oriented biological macromolecules)”, Ultramicroscopy, vol.13, pp.165-183 (1984)) and non-patent Reference 4 (van Heel M., “Classification of very large electron microscopical image data sets”, Optik, vol. 82, pp. 114-126 (1989)).

The averaging processing unit 19 averages the pixel values of the plurality of derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) for each group. By doing so, a two-dimensional averaged image is generated for each group. This averaging is not particularly limited as long as it is an averaging (for example, arithmetic averaging) that can remove random noise.

  Note that the averaging processing unit 19 may weight the pixel value of the derived image with a coefficient corresponding to the similarity of the derived image. The higher the degree of similarity of the derived image, the larger the coefficient to be weighted to the pixel value of the derived image, while the lower the degree of similarity of the derived image, the more the coefficient to be weighted to the pixel value of the derived image. It is preferable to set a small value. Thereby, the quality of the two-dimensional averaged image can be improved. If a lookup table (not shown) indicating the correspondence between the similarity value and the weighting coefficient value is prepared in advance, the averaging processing unit 19 refers to the lookup table to determine the similarity. It is possible to set an optimum weighting coefficient according to the response.

The image classification unit 18 may change the width Δ between the lower limit S _max −Δ and the upper limit S _max according to the S / N ratio of each particle image or the entire image. That is, it is preferable to decrease Δ as the S / N ratio of each particle image or the entire image is higher, and to increase Δ as the S / N ratio of the particle image is lower. As a result, the quality of the two-dimensional averaged image generated by the averaging processing unit 19 can be improved. If a lookup table (not shown) indicating the correspondence between the S / N ratio of the particle image and the value of the width Δ is prepared in advance, the image classification unit 18 refers to the lookup table to It is possible to set an optimum width Δ according to the S / N ratio of the particle image or the entire image.

Update processing unit 20 has a function of updating the contents stored in the image storage unit 161 by replacing the reference image R ₁ to R _N containing the reference particle images in the data of the two-dimensional averaging image. That is, the update processing unit 20 determines whether good or not than the averaging process unit 19 two-dimensional averaging image group reference quality image R ₁ in to R _N generated by. This determination is performed by comparing the S / N ratio and the contour extraction result of both images. When the update processing unit 20 determines that the quality of the two-dimensional averaged image group is better than that of the reference images R _{1 to} R _N , the update processing unit 20 replaces the reference image group with the two-dimensional averaged image group. Thereby, the peak detection unit 12 can detect the peak of the similarity distribution with higher accuracy by using the updated reference images R _{1 to} R _P (P is a positive integer).

(Particle structure analysis processing)
The operation of the particle structure analysis apparatus 1 having the above configuration will be described below. FIG. 4 is a flowchart schematically showing an example of the procedure of the particle structure analysis process performed by the particle structure analysis apparatus 1. In the process of FIG. 4, the process of updating the stored contents of the reference image storage unit 162 is not executed.

Referring to FIG 4, in step S1, the partial image extraction unit 11 reads the entire image including a plurality of target particle image from the image storage unit 161, followed by, extracts the partial image P ₁ to P _M from the entire image (Step S2). Among these partial images P _{1 to} P _M, the partial images P _i and P _{i + 1} adjacent in a predetermined direction (for example, the X-axis direction) overlap each other so as to include the same target particle image.

Figure 5 is a diagram schematically showing a part Pa~Pf of the whole image 30 and the partial image _P 1 to P _M. As shown in FIG. 5, the partial images Pa to Pc are extracted from the entire image 30 so as to overlap each other in the X-axis direction. The same applies to the partial images Pd to Pf. Therefore, the partial images Pa to Pc include the same target particle image 31 to be detected, and the partial images Pd to Pf include the same target particle image 32 to be detected. The positions of the partial images Pa, Pb, and Pc are specified by vectors Va, Vb, and Vc based on the origin O of the effective area of the entire image 30, respectively.

Referring now to FIG. 4, in step S3, the peak detector 12, a number i representing the partial image P _i to be processed is set to an initial value "1" (step S3), and the peak detection processing Start. The peak detection unit 12 reads the reference images R _{1 to} R _N from the reference image storage unit 162, and the similarity distribution DS _{i, k} (x, y) between the reference images R _{1 to} R _N and the partial image P _i. calculates a theta) (step S4), and then, the similarity distribution _{DS i, k (x, y} , from theta), one or more peaks having more similarity threshold Pt (i, 1) ~Pt ( i, Q) (Q is a positive integer) is detected (step S5).

However, when the particle image detection unit 2 operates in the processing mode based on the MRP method, the peak detection unit 12 obtains the maximum similarity from the similarity distribution DS _{i, k} (x, y, θ) according to an instruction from the controller 22. Only a single peak Pt (i, 1) is detected. On the other hand, when the particle image detection unit 2 operates in the processing mode based on the MRMP method, the peak detection unit 12 follows the instructions from the controller 22 from the similarity distribution DS _{i, k} (x, y, θ) to a plurality of peaks Pt ( i, 1) to Pt (i, Q) can be detected.

In the next step S6, the vector calculating unit 13, the partial image _P a predetermined reference position in the local region of the _i (partial image _{P i} of the center) from the peak Pt (i, 1) the position of ~Pt (i, Q) Position setting vectors V _i (1) to V _i (Q) are calculated.

Next, the vector calculation unit 13 determines whether or not the vector calculation process has been completed for all of the partial images P _{1 to} P _M (step S7). If the vector calculation process has not been completed for all partial images P _{1 to} P _M (NO in step S7), the number i is incremented (step S8), and steps S4 to S7 are repeatedly executed.

On the other hand, when the vector calculation process has been completed for all the partial images P _{1 to} P _M (YES in step S7), the peak distribution generation unit 14 performs the position setting vector V _i (j ) (I is an integer from 1 to M, j is an integer from 1 to Q), and the position of the peak Pt (i, j) is set at a position specified by (2) to generate a two-dimensional peak distribution (step S9). Thereafter, the peak distribution generation unit 14 records this two-dimensional peak distribution in the peak distribution image storage unit 163 (step S10).

FIGS. 6A and 6B are diagrams illustrating reference images R ₁ and R ₂ including the same reference particle image when viewed from different directions, respectively. Figure 7 (A) ~ (C) is a diagram showing FIG. 6 (B) of the reference image _{R 2} and the partial image Pa in FIG. 5, Pb, Pc and positioning vector is calculated based on the VSa, VSb, the VSc It is. 7A to 7C, a peak is detected at a substantially central position of the target particle image 31 in all. For this reason, the position setting vectors VSa, VSb, and VSc are vectors from the centers of the partial images Pa, Pb, and Pc to the approximate center position of the target particle image 31, respectively.

  In this case, the peak distribution generation unit 14 shifts the partial images Pa, Pb, and Pc by the position setting vectors VSa, VSb, and VSc, respectively, within the entire image area. As a result, as shown in FIGS. 7A to 7C, the partial images Pa, Pb, and Pc are shifted to partial images SPa, SPb, and SPc, respectively. Then, the peak distribution generation unit 14 sets the peak position at the reference position (for example, the center) of the shifted partial images SPa, SPb, SPc within the entire image area (step S9).

  After the two-dimensional peak distribution is recorded in step S10, the particle image search unit 15 reads this two-dimensional peak distribution from the peak distribution image storage unit 163, and the peak density and peak value of the two-dimensional peak distribution (the above-mentioned The position of each target particle image is detected by searching for the target particle image using the similarity or cumulative similarity) (step S11).

  FIG. 8 is a diagram schematically illustrating an example of the two-dimensional peak distribution 50. As shown in FIG. 8, a plurality of peaks Pt, Pt,... Appear in the two-dimensional peak distribution 50. As illustrated in FIG. 8, the particle image search unit 15 sequentially calculates the peak density in the search window while scanning the search window 51 with respect to the two-dimensional peak distribution 50. At the same time, when the positions of a plurality of peaks Pt appearing in the two-dimensional peak distribution 50 overlap, the particle image search unit 15 adds the similarities of the plurality of peaks to obtain a cumulative similarity. The particle image search unit 15 (i) when the peak density is equal to or higher than the first threshold, or (ii) the similarity or cumulative similarity of the peak position in the search window is equal to or higher than the second threshold. Sometimes, it is determined that the target particle image exists at the position of the search window 51, and the determination result is given to the particle image extraction unit 17.

  Thereafter, the particle image extraction unit 17 reads the entire image from the image storage unit 161, and extracts an image region where the target particle image exists as a particle image from the entire image based on the search result by the particle image search unit 15 ( Step S12). The particle image group is transferred to the three-dimensional structure analysis unit 3 in FIG.

  The three-dimensional structure analyzing unit 3 estimates the three-dimensional structure of the target particle image based on the particle image group transferred from the particle image detecting unit 2 (step S13). This completes the particle structure analysis process.

  The effects produced by the particle structure analysis process are as follows.

As described above, the peak of the similarity distribution between each partial image P _i and the reference particle image is detected, and the position setting vector from the reference position to the peak position in each partial image P _i is calculated. . Then, a two-dimensional peak distribution is generated by setting the peak position at the position specified by the position setting vector. Since the position setting vector is a vector indicating a region where the existence probability of the target particle image is relatively high, in the two-dimensional peak distribution, the peak is likely to be concentrated at the position of the target particle image. Therefore, it is possible to detect the target particle image with high accuracy and in a short time based on the two-dimensional peak distribution even from the entire image having a low S / N ratio.

  In addition, since the particle image search unit 15 searches for the target particle image based on the peak density and cumulative similarity of the two-dimensional peak distribution, it is possible to detect the target particle image with high accuracy. Further, the particle image search unit 15 sequentially calculates the number of peaks in the search window as the peak density while scanning the search window with respect to the two-dimensional peak distribution. Therefore, the position of the target particle image can be automatically detected by sequentially calculating the peak density and the cumulative similarity at a high speed with a small amount of calculation.

  In particular, when the particle image detection unit 2 is operated in the processing mode by the MRMP method, a plurality of peaks are likely to appear in a region where the target particle image is buried in noise in the entire image having a low S / N ratio. Since the MRMP method is a method for detecting a plurality of peaks for each partial image and detecting a target particle image using a plurality of peaks as described above, the target particle is detected from the entire image having a very low S / N ratio. An image can be detected with high accuracy.

  Therefore, the three-dimensional structure analysis unit 3 can estimate the three-dimensional structure of the target particle with high accuracy.

  In order to improve the detection accuracy of target particles, the partial image extraction unit 11 may include a bandpass filter that extracts only the spatial frequency component of a specific band from the entire image or the spatial frequency component of the partial image. Good. If this bandpass filter is used, the target particle image can be detected from the spatial frequency component of a specific band. For example, a high-pass filter can be used to remove noise that exists only in high-band components from the entire image, or a low-pass filter can be used to obtain a uniform non-uniform luminance distribution over the entire target particle image or the entire electron microscope image. By making the luminance distribution, it is possible to improve the detection accuracy of the target particle image.

  Alternatively, the partial image extraction unit 11 may have a filter bank that outputs a plurality of spatial frequency components in different bands. When a filter that performs discrete wavelet transform is used, spatial frequency components of a plurality of different bands can be obtained from one whole image. If a two-dimensional peak distribution is generated based on the spatial frequency components of a plurality of different bands, it is possible to select an optimal two-dimensional peak distribution for detecting the target particle image from these two-dimensional peak distributions. .

  In order to improve the detection accuracy of the target particles, the peak distribution generation unit 14 extracts a spatial frequency component of a specific band from the spatial frequency components of the two-dimensional peak distribution, or a plurality of different band spaces. You may have the filter which can output a frequency component. By using such a filter, it is possible to obtain a spatial frequency component in an optimum band for detection of the target particle image according to the quality of the entire image.

(Repeat process)
FIG. 9 is a flowchart showing another procedure of the particle structure analysis process performed by the particle structure analysis apparatus 1. Steps S1 to S13 in the flowchart of FIG. 9 are the same as steps S1 to S13 in the flowchart of FIG. In the flowchart of FIG. 9, steps S20 and S21 are newly added in order to update the stored contents of the reference image storage unit 162.

  Referring to FIG. 9, in step S <b> 12, as described above, the particle image extraction unit 17 reads the entire image from the image storage unit 161, and based on the search result by the particle image search unit 15, the target particle is read from the entire image. An image region where an image exists is extracted as a particle image. The particle image is output to the image classification unit 18.

  In the next step S20, the controller 22 determines whether or not to end the repetition process. If the repetitive process is not completed (NO in step S20), the feedback process in step S21 is executed. FIG. 10 is a flowchart schematically showing the procedure of feedback processing.

Referring to FIG. 10, in step S220, the image classification unit 18 uses a plurality of derived images S (i, 0) to S (i, P _i ) from each particle image S (i) input from the particle image extraction unit 17. ) Is generated. FIGS. 11A, 11B, and 12 are diagrams for explaining a method of generating the derived images S (i, 0) to S (i, P _i ).

As shown in FIG. 11A, based on each particle image S (i), P derived images S (i, 0) to S (i, P _i corresponding to P peak similarities, respectively. = P) is generated. Mathematically, the p-th derived image S (i, p) has particles in the X-axis translation direction, the Y-axis translation direction, and the angle θ direction by amounts determined by the parameters xp, yp, θ = thp, respectively. It is an image generated by moving the image S (i). As shown in FIG. 11B, xp represents the amount of movement in the X-axis translation direction, yp represents the amount of movement in the Y-axis translation direction, and thp represents the amount of rotational movement around the origin O, respectively. . Here, although the number of derived images is plural for each particle image S (i), the number is not limited to this, and the number of derived images may be one for each particle image S (i). .

FIG. 12 is a schematic diagram for explaining an example of the relationship between the movement amount (X-axis translational movement amount, Y-axis translational movement amount and rotational movement amount) of the particle image S (i) and the peak similarity. In the graph of FIG. 12, a plurality of peaks appear according to the amount of movement. Of these peaks, the image classification unit 18 selects only the peaks SP _max and SP having a similarity within a range in which the maximum similarity S _max is the upper limit and S _max −Δ is the lower limit. The image classification unit 18 generates derived images S (i, 0) to S (i, P _i ) corresponding to the peaks within such a range (step S220).

Derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ) are generated for all particle images S (1) to S (T). After that, in step S221, the image classifying unit 18 obtains the derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P) from the derived image storage unit 164. _T ) is read. Further, the image classification unit 18 vectorizes these derived images S (1, 0) to S (1, P ₁ ),..., S (T, 0) to S (T, P _T ), and classifies them into a plurality of groups. To do.

Next, the averaging processing unit 19 generates a two-dimensional averaged image for each group by averaging the pixel values of the plurality of classified derived images for each group (step S222). Then, the update processing unit 20 updates the stored contents of the image storage unit 161 by replacing the data of the reference image R ₁ to R _N containing the reference particle images in the data of the two-dimensional averaging image group (step S223). Here, the update processing unit 20, the quality of the 2 dimensional averaging image group only if than that of the reference image R ₁ to R _N is good, the reference image R ₁ to R _N two-dimensional averaging image You may replace it with a group. Thereafter, the controller 22 returns the process to the flowchart of FIG. 9 and repeatedly executes the procedures of steps S3 to S20.

  When the number of executions of the iterative process reaches a predetermined number, or when the detection accuracy of the target particle image reaches a certain accuracy, the controller 22 determines that the iterative process is to be ended (YES in step S20), and image classification. The operations of the unit 18, the averaging processing unit 19 and the update processing unit 20 are stopped. Then, the three-dimensional structure analysis unit 3 estimates the three-dimensional structure of the target particle image based on the particle image group input from the particle image detection unit 2 (step S13). This completes the particle structure analysis process.

  The effects of the above repeated processing are as follows.

In the above iterative process, the reference images R _{1 to} R _N are replaced with a two-dimensional averaged image with an improved S / N ratio (steps S222 and S223). Therefore, it is possible to improve the detection accuracy of the target particle image by repeatedly executing steps S3 to S12 and S20 to S21 of FIG.

  In particular, in the feedback process (FIG. 10), a plurality of derived images are generated from the particle image, and these derived images are classified into a plurality of groups (steps S220 to S221). Therefore, the quality of the two-dimensional averaged image can be improved, and the detection accuracy of the target particle image can be further increased.

(Evaluation by test)
Next, the execution result of the operation test of the particle structure analysis apparatus 1 will be described. The function of the particle structure analysis apparatus 1 was realized by a computer program, and the test was executed by executing this computer program.

  FIG. 13 is a diagram illustrating reference images 100 to 103 including reference particle images (model particle images) used in the test. As shown in FIG. 13, four reference images 100, 101, 102, and 103 were generated by rotating the reference particle image of the reference image 100 by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. . These four reference images 100 to 103 were randomly selected and distributed at random positions in a two-dimensional image of 5120 pixels × 5120 pixels. Furthermore, the entire image 30 including 100 or 200 target particle images is generated by superimposing noise on the two-dimensional image.

  14A shows the target particle image in the entire image 30 with an SN ratio (= S / N ratio) of 0.1, and FIG. 14B shows the target particle in the entire image 30 with an SN ratio of 0.01. FIG. 14B is a diagram showing the target particle images in the entire image 30 with an SN ratio of 0.001.

  The target particle image was detected from the entire image 30 by the first operation mode (FIG. 4) using the MRP method and the MRMP method. FIG. 15A is a diagram showing the entire image 30, and FIG. 15B is an enlarged view of a partial area Wd of the entire image 30. As shown in FIG. 15B, a region surrounded by a white line frame indicates a target particle image detected (picked up) using the MRMP method.

  FIG. 16 is a graph showing the S / N ratio dependence of the pickup efficiency when 100 target particle images are picked up. The horizontal axis of this graph represents the SN ratio (= S / N ratio) of the entire image 30, and the vertical axis of the graph represents the picking efficiency. Here, “pickup efficiency” is the coincidence rate (%) between the actual position of the model particle image in the entire image 30 and the position of the detected target particle image. For example, when 50 positions out of 100 detected target particle images coincide with the actual positions of the model particle images, the pick-up efficiency is 50%. In the graph of FIG. 16, the symbol “x” indicates the average pick-up efficiency by the MRP method, and the symbol “◯” indicates the average pick-up efficiency by the MRMP method. The solid line connects the points indicating the average pick-up efficiency by the MRP method. In this graph, although it is difficult to discriminate, the pick-up efficiency values of the SN ratios of 1, 0.1, and 0.01 were almost the same between the MRP method and the MRMP method.

  FIG. 17 is a graph showing the S / N ratio dependence of the pickup efficiency when 200 target particle images are picked up. The horizontal axis of this graph represents the SN ratio (= S / N ratio) of the entire image 30, and the vertical axis of the graph represents the picking efficiency. For example, when 100 positions among the detected positions of the 200 target particle images coincide with the actual positions of the model particle images, the pick-up efficiency is 50%. In the graph of FIG. 17, the solid line indicates the average pick-up efficiency by the MRP method, and the symbol “◯” indicates the average pick-up efficiency by the MRMP method for the entire image 30 with the SN ratio of 0.001.

According to the graphs of FIGS. 16 and 17, it can be seen that the target particle image is picked up with an efficiency of 100% from the entire image 30 having a low S / N ratio of 10 ⁻² order. In particular, when the target particle image is picked up using the MRMP method, it is understood that the picking efficiency is improved as compared with the case where the target particle image is picked up using the MRP method.

  FIG. 18 is a graph showing the parameter dependence of the pick-up efficiency when 100 target particle images are picked up from the entire image 30 having an S / N ratio of 0.001. The horizontal axis of this graph represents a combination (parameter) of the radius R of the search window 51 of FIG. 8 and the threshold value ρ with respect to the peak density. When the peak density is equal to or higher than the threshold ρ, it is determined that the target particle image is located at the position of the search window 51. On the other hand, FIG. 19 is a graph showing the parameter dependence of the pick-up efficiency when 200 target particle images are picked up from the entire image 30 having an S / N ratio of 0.001. The horizontal axis of this graph also indicates a combination (parameter) of the radius R of the search window 51 in FIG. 8 and the threshold value ρ with respect to the peak density. 18 and 19 both show the pick-up efficiency by the MRP method and the pick-up efficiency by the MRMP method.

  According to the graphs of FIGS. 18 and 19, it can be seen that the pick-up efficiency of about 100% can be achieved by adjusting the parameters in both the MRP method and the MRMP method. It can also be seen that when the MRMP method is used, the dependency of the pick-up efficiency on the set value of the parameter is lower than when the MRP method is used. In other words, the MRMP method has excellent noise resistance with respect to parameters and has high robustness against parameter variations.

  FIG. 20 is a graph showing a cumulative frequency distribution of the target particle images picked up when 100 target particle images are picked up from the entire image 30 having an S / N ratio of 0.001. FIG. 21 is a graph showing a cumulative frequency distribution of target particle images picked up when 200 target particle images are picked up from the entire image 30 having an S / N ratio of 0.001. 20 and 21 are composed of a total of 16 graphs when different parameters (combination of radius R and threshold value ρ) are used, and each graph shows the value of the parameter used. The vertical axis of each graph indicates the cumulative frequency of the picked up target particle image, and the horizontal axis of each graph shows the actual position of the model particle image in the entire image 30 and the picked up target particle image. The distance (unit: pixel) between these positions is shown. The horizontal axis of each graph represents the position P (i) of the i-th model particle image among 100 or 200 model particle images, and the position P (i) of the 100 or 200 target particle images picked up. ) Represents the deviation from the position Q (i) of the closest target particle image. In each graph, the broken line indicates the result when the MRP method is used, and the solid line indicates the result when the MRP method is used.

  In the graphs of FIG. 20 and FIG. 21, the larger the slope of the rising portion of the curve when the cumulative frequency converges from 0 to 100, the smaller the positional deviation of the picked up target particle image. Is expensive. 20 and 21, it can be seen that high detection efficiency can be achieved by adjusting parameters in both the MRP method and the MRMP method. In addition, when the MRMP method is used, the convergence of the accumulated frequency is generally better regardless of the parameter setting values than when the MRP method is used. From these graphs, it can be seen that the MRMP method has excellent noise resistance with respect to parameters and high robustness against parameter fluctuations.

  FIGS. 22 and 23 are graphs obtained when an electron microscope including an observed particle image of a protein called TRPM2 is used as the entire image 30. FIG. FIG. 22 is a graph showing the relationship between the number of picked-up particles (number of picked-up observed particle images) and the correlation value, and FIG. 23 shows the frequency distribution of the picked-up observation particle images. Each graph shows the value of the parameter used (combination of radius R and threshold ρ). In each graph, the broken line indicates the result when the MRP method is used, the solid line indicates the result when the MRP method is used, and the dotted line indicates the result when the observed particle image is visually identified. .

  The correlation value in the graph of FIG. 22 is a cross-correlation coefficient between the observed particle image picked up (detected) and the projected image of the three-dimensional structure of the TRPM2 protein. The graph of FIG. 22 means that the higher the correlation value, the higher the detection accuracy of the observed particle image. According to the graph of FIG. 22, it can be seen that when the MRP method and the MRMP method are used, if the parameters are adjusted, higher detection accuracy can be achieved than when the observed particle image is detected visually. For example, (R = 1, ρ = 1.00), (R = 2, ρ = 0.50), (R = 2, ρ = 1.00), (R = 3, ρ = 0.67), When the parameters (R = 5, ρ = 0.40) are used, higher detection accuracy is obtained than when the observed particle image is detected visually.

  Further, according to the graph of FIG. 22, it can be seen that when the MRMP method is used, the parameter dependence of the detection accuracy of the observed particle image is lower than when the MRP method is used. Therefore, it can be seen that the MRMP method has excellent noise resistance with respect to parameters and has high robustness against parameter variations.

  The correlation value in the graph of FIG. 23 is a cross-correlation coefficient between the observed particle image picked up (detected) and the projected image of the three-dimensional structure of the TRPM2 protein. As the bias of the frequency distribution shifts to a higher correlation value, the detection accuracy of the observed particle image is higher. According to the graph of FIG. 23, it can be seen that in general, when the MRP method and the MRMP method are used, higher detection accuracy can be achieved than when the observed particle image is detected visually.

As mentioned above, although embodiment of this invention was described with reference to drawings, these are the illustrations of this invention, Various structures other than the above are also employable. For example, in the above embodiment, the entire image input to the particle structure analysis apparatus 1 is an electron microscope image, and the reference images R _{1 to} R _N stored in the reference image storage unit 162 are modeled on biopolymer images. However, the present invention is not limited to this. The entire image may be a low S / N ratio image including a target particle image other than the biopolymer image, or the reference images R _{1 to} R _N are images imitating a target particle image other than the biopolymer image. It may be. The particle structure analysis apparatus 1 of the above embodiment can detect an image of a structure such as a building as a target particle image from a satellite image taken from the ground, for example.

Moreover, the process of the particle image detection part 2 can be parallelized. For example, by using a plurality of processors, the similarity calculation process in the peak detection unit 12 can be parallelized for each reference image R _k (k is an arbitrary integer from 1 to N) or for each partial image.

It is a figure for demonstrating the outline | summary of a single particle analysis method. 1 is a functional block diagram showing a schematic configuration of a particle structure analysis apparatus which is an image processing system according to an embodiment of the present invention. It is a functional block diagram which shows schematic structure of a particle image detection part. It is a flowchart which shows the procedure of a particle structure analysis process roughly. It is a figure which shows typically a whole image and a part of partial image. (A), (B) is a figure which illustrates the reference image containing the same reference particle image when it sees from a different direction, respectively. (A)-(C) is a figure which shows the position setting vector calculated based on the reference image of FIG.6 (B), and the partial image of FIG. It is a figure which shows a two-dimensional peak distribution typically. It is a flowchart which shows the procedure of the particle | grain structure analysis process accompanied by a repetition process. It is a flowchart which shows the procedure of a feedback process roughly. It is a figure for demonstrating the production | generation method of a derivative image. It is the schematic for demonstrating an example of the relationship between a movement amount and peak similarity. It is a figure which shows the reference image for a test. (A), (B), (C) is a figure which shows the model particle image (target particle image) which has SN ratio (S / N ratio) of 0.1, 0.01, and 0.001, respectively. . (A) is a figure which shows the whole image, (B) is the figure which expanded the partial area | region Wd of the whole image. It is a graph which shows the S / N ratio dependence of the pick-up efficiency of a target particle image. It is a graph which shows the S / N ratio dependence of the pick-up efficiency of a target particle image. It is a graph which shows the parameter dependence of the picking-up efficiency of an object particle image. It is a graph which shows the parameter dependence of the picking-up efficiency of an object particle image. It is a graph which shows the cumulative frequency distribution of the object particle image picked up. It is a graph which shows the cumulative frequency distribution of the object particle image picked up. It is the graph obtained from the electron microscope image containing the observation particle image of TRPM2 protein. It is the graph obtained from the electron microscope image containing the observation particle image of TRPM2 protein.

Explanation of symbols

1 Particle structure analyzer (image processing system)
DESCRIPTION OF SYMBOLS 2 Particle image detection part 3 Three-dimensional structure analysis part 10 Image input part 11 Partial image extraction part 12 Peak detection part 13 Vector calculation part 14 Peak distribution generation part 15 Particle image search part 150 Peak density calculation part 151 Position detection part 16 Data storage part 161 Image storage unit 162 Reference image storage unit 163 Peak distribution image storage unit 164 Derived image storage unit 17 Particle image extraction unit 18 Image classification unit 19 Averaging processing unit 20 Update processing unit 21 Image memory 22 Controller 30 Whole image 31, 32 Object Particle image

Claims (29)

A partial image extraction unit that extracts a plurality of partial images from an entire image including a plurality of target particle images;
An image storage unit storing image data of a reference particle image simulating the target particle image;
A peak detector for calculating a similarity distribution between each partial image and the reference particle image for each partial image, and detecting a peak of the similarity distribution;
A vector calculation unit for calculating a position setting vector from the predetermined reference position in each partial image to the peak position for each partial image;
A peak distribution generation unit configured to generate a two-dimensional peak distribution by setting a position of the peak at a position specified by the position setting vector with respect to the region of the entire image;
With
The partial image extraction unit is an image processing system that extracts the plurality of partial images from the whole image so that partial images adjacent in a predetermined direction among the plurality of partial images overlap each other.   The image processing system according to claim 1, wherein the peak distribution generation unit shifts the predetermined reference position in each partial image by the position setting vector calculated for each partial image, and An image processing system, wherein the shifted reference position is a position specified by the position setting vector.   3. The image processing system according to claim 1, wherein the peak detection unit detects at least one peak having a similarity equal to or higher than a threshold value for each of the partial images.   The image processing system according to claim 1, wherein the peak detection unit sequentially selects a target pixel area from the partial images, and the reference particle image and the target pixel area for each target pixel area. An image processing system that obtains the similarity distribution by calculating a similarity with an image region including the image area.   5. The image processing system according to claim 1, further comprising a particle image search unit configured to detect a position of the target particle image based on the two-dimensional peak distribution.   6. The image processing system according to claim 5, wherein the particle image search unit searches for a region where the density of peaks appearing in the two-dimensional peak distribution is equal to or greater than a threshold value, and based on the search result, the target particle image. Image processing system that detects the position of The image processing system according to claim 6,
The particle image search unit includes:
A peak density calculator that sequentially calculates the density based on the number of peaks in the search window while scanning the search window with respect to the two-dimensional peak distribution;
A position detection unit that determines that the target particle image is positioned at the position of the search window when the density is equal to or higher than a threshold;
Including an image processing system. The image processing system according to any one of claims 5 to 7,
The particle image search unit searches for a region where the similarity of the peak position appearing in the two-dimensional peak distribution is equal to or greater than a threshold, and detects the position of the target particle image based on the search result,
An image processing system in which, when positions of a plurality of peaks appearing in the two-dimensional peak distribution overlap, a position of the target particle image is detected based on a result of adding similarities of the plurality of peaks. The image processing system according to any one of claims 5 to 8,
A particle image extraction unit that extracts a particle image including a target particle image at a position detected by the particle image search unit from the entire image;
An image classification unit for classifying the particle image or a derived image derived from the particle image into a plurality of groups;
An averaging processor that averages the classified particle image or derivative image for each group to generate a two-dimensional averaged image;
An update processing unit that updates the storage content of the image storage unit by replacing the image data of the reference particle image with the data of the two-dimensional averaged image;
Further comprising
The partial image extraction unit, the peak detection unit, the vector calculation unit, the peak distribution generation unit, and the particle image search unit again perform processing using the particle image in the two-dimensional averaged image as the reference particle image. An image processing system to be executed.   The image processing system according to claim 9, wherein the partial image extraction unit, the peak detection unit, the vector calculation unit, the peak distribution generation unit, the particle image search unit, the particle image extraction unit, and the averaging The processing unit and the update processing unit perform an iterative process.   11. The image processing system according to claim 9, wherein the image classification unit classifies the particle image or the derived image into a plurality of groups according to a direction in which particles are directed.   12. The image processing system according to claim 1, wherein the whole image is an image obtained by imaging a plurality of biopolymers included in a sample with an electron microscope.   The image processing system according to claim 1, further comprising a three-dimensional structure estimation unit that estimates a three-dimensional structure of the target particle image based on the particle image. (A) extracting a plurality of partial images from an entire image including a plurality of target particle images;
(B) Reading image data of a reference particle image simulating the target particle image from an image storage unit, calculating a similarity distribution between each partial image and the reference particle image for each partial image, and calculating the similarity Detecting a peak of the degree distribution;
(C) calculating a position setting vector from the predetermined reference position in each partial image to the peak position for each partial image;
(D) generating a two-dimensional peak distribution by setting the position of the peak at a position designated by using the position setting vector with respect to the region of the entire image;
With
In the step (a), the plurality of partial images are extracted from the whole image so that partial images adjacent in a predetermined direction among the plurality of partial images overlap each other.   15. The image processing method according to claim 14, wherein in the step (b), at least one peak having a similarity equal to or higher than a threshold is detected for each partial image. The image processing method according to claim 14, comprising:
(E) detecting a position of the target particle image based on the two-dimensional peak distribution;
An image processing method further comprising:   17. The image processing method according to claim 16, wherein the step (e) searches for a region where the density of peaks appearing in the two-dimensional peak distribution is equal to or greater than a threshold, and the target particle image is based on the search result. An image processing method including a step of detecting a position of the image. The image processing method according to claim 17, comprising:
The step (e)
Sequentially scanning the density based on the number of peaks in the search window while scanning the search window for the two-dimensional peak distribution;
Determining that the target particle image is located at the position of the search window when the density is equal to or higher than a threshold;
Including an image processing method. The image processing method according to any one of claims 16 to 18, comprising:
The step (e) includes a step of searching for a region where the similarity of the position of the peak appearing in the two-dimensional peak distribution is a threshold value or more, and detecting the position of the target particle image based on the search result,
An image processing method in which, when positions of a plurality of peaks appearing in the two-dimensional peak distribution overlap, a position of the target particle image is detected based on an addition result of similarities of the plurality of peaks. An image processing method according to any one of claims 16 to 19, comprising:
(F) extracting a particle image including the target particle image at a position detected by searching the target particle image from the whole image;
(G) Classifying the particle images or derived images derived from the particle images into a plurality of groups, and averaging the classified particle images or derived images for each group to generate a two-dimensional averaged image. When,
(H) updating the storage content of the image storage unit by replacing the image data of the reference particle image with the data of the two-dimensional averaged image;
Further comprising
The image processing method in which the steps (a) to (d) are performed again using a particle image in the two-dimensional averaged image as the reference particle image.   21. The image processing method according to claim 20, wherein the steps (a) to (h) are repeatedly executed. A program that is read from a computer-readable recording medium and causes a computer to execute image processing,
The image processing
A partial image extraction process for extracting a plurality of partial images from an entire image including a plurality of target particle images;
Image data of a reference particle image that imitates the target particle image is read from an image storage unit, a similarity distribution between each partial image and the reference particle image is calculated for each partial image, and the similarity distribution A peak detection process for detecting peaks;
A vector calculation process for calculating a position setting vector from the predetermined reference position in each partial image to the peak position for each partial image;
A peak distribution generation process for generating a two-dimensional peak distribution by setting the position of the peak at a position specified by using the position setting vector with respect to the region of the entire image;
Including
In the partial image extraction process, the plurality of partial images are extracted from the whole image so that partial images adjacent in a predetermined direction among the plurality of partial images overlap each other.   23. The program according to claim 22, wherein the image processing further includes a particle image search process for detecting a position of the target particle image based on the two-dimensional peak distribution. The program according to claim 23, wherein
The image processing is
A particle image extraction process for extracting, from the entire image, a particle image including a target particle image at a position detected by the particle image search process;
Image classification processing for classifying the particle image or a derived image derived from the particle image into a plurality of groups;
An averaging process for averaging the classified particle image or derivative image for each group to generate a two-dimensional averaged image;
An update process for updating the storage content of the image storage unit by replacing the image data of the reference particle image with the data of the two-dimensional averaged image;
Further including
The partial image extraction process, the peak detection process, the vector calculation process, and the peak distribution generation process are executed again using a particle image in the two-dimensional averaged image as the reference particle image.   25. The program according to claim 24, wherein the partial image extraction process, the peak detection process, the vector calculation process, the peak distribution generation process, the particle image search process, the particle image extraction process, the averaging process, and The update process is executed repeatedly. A recording medium recording a program for causing a computer to execute image processing,
The image processing is
A partial image extraction process for extracting a plurality of partial images from an entire image including a plurality of target particle images;
Image data of a reference particle image that imitates the target particle image is read from an image storage unit, a similarity distribution between each partial image and the reference particle image is calculated for each partial image, and the similarity distribution A peak detection process for detecting peaks;
A vector calculation process for calculating a position setting vector from the predetermined reference position in each partial image to the peak position for each partial image;
A peak distribution generation process for generating a two-dimensional peak distribution by setting the position of the peak at a position specified by using the position setting vector with respect to the region of the entire image;
Including
In the partial image extraction process, the plurality of partial images are extracted from the whole image so that partial images adjacent in a predetermined direction among the plurality of partial images overlap each other.   27. The recording medium according to claim 26, wherein the image processing detects a position of the target particle image by searching the target particle image based on a density of peaks appearing in the two-dimensional peak distribution. A recording medium further including a search process. The recording medium according to claim 27, wherein
The image processing
A particle image extraction process for extracting, from the entire image, a particle image including a target particle image at a position detected by the particle image search process;
Image classification processing for classifying the particle image or a derived image derived from the particle image into a plurality of groups;
An averaging process for averaging the classified particle image or derivative image for each group to generate a two-dimensional averaged image;
An update process for updating the storage content of the image storage unit by replacing the image data of the reference particle image with the data of the two-dimensional averaged image;
Further including
The recording medium in which the partial image extraction process, the peak detection process, the vector calculation process, and the peak distribution generation process are performed again using a particle image in the two-dimensional averaged image as the reference particle image.   29. The recording medium according to claim 28, wherein the partial image extraction process, the peak detection process, the vector calculation process, the peak distribution generation process, the particle image search process, the particle image extraction process, and the averaging process. And a recording medium in which the update process is repeatedly executed.