JP7084653B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Patent Document 1 and Non-Patent Document 1 disclose a technique for reconstructing a three-dimensional image of a sample based on a projected image of the sample. In this technique, a three-dimensional image of a sample is reconstructed by arranging a concentration quantum obtained by quantizing a concentration value.

Japanese Patent No. 5427133

"Nonlinear Reconstruction Method Using Concentration Quantum", Norio Baba, Kenji Kaneko: Microscope, Vol. 53, 2018

However, in the above-mentioned prior art, a streak-like false image may occur in the reconstructed three-dimensional image, and the three-dimensional image may not be reconstructed with high accuracy.

The present invention provides an image processing device, an image processing method, and an image processing program capable of accurately reconstructing a three-dimensional image as compared with the case of reconstructing a three-dimensional image simply by arranging density quanta. The purpose is to provide.

In order to solve the above problems, the invention of the image processing apparatus according to the first aspect includes an acquisition unit that acquires a plurality of first projection images at a plurality of angles with respect to a sample, and the plurality of first projection images and reconstruction. A reconstructed image is created by arranging a calculation unit that calculates an error distribution based on a plurality of second projection images at the plurality of angles of the image and a density quantum obtained by quantizing the density value based on the error distribution. It includes a generation unit to be generated, and a rearrangement unit for rearranging the density quantum based on the generated reconstruction image and the error distribution.

In the invention of the image processing apparatus according to the second aspect, the rearrangement unit is based on the error distribution, and the density quantum is changed from the pixel having the largest number of density quanta to the pixel having the least number of density quanta. The concentration quantum is rearranged by moving.

The invention of the image processing apparatus according to the third aspect is a control unit that controls the processing by the calculation unit, the generation unit, and the rearrangement unit to be repeated while gradually increasing the resolution of the reconstructed image. To prepare for.

The invention of the image processing method according to the fourth aspect is a step in which a computer acquires a plurality of first projection images at a plurality of angles with respect to a sample, and the plurality of the plurality of first projection images and reconstruction images. A step of calculating an error distribution based on a plurality of second projection images at an angle, and a step of generating a reconstructed image by arranging a concentration quantum representing a concentration unit based on the error distribution, were generated. A process including the step of rearranging the concentration quantum based on the reconstruction image and the error distribution is performed.

The invention of the image processing program according to the fifth aspect is a program for making a computer function as each part of the image processing apparatus according to any one of the first to third aspects.

According to the present invention, there is an effect that the three-dimensional image can be reconstructed with high accuracy as compared with the case where the three-dimensional image is reconstructed simply by arranging the density quanta.

It is a block diagram of a three-dimensional image reconstruction system. It is a flowchart of image processing. It is a figure for demonstrating the rearrangement of a concentration quantum. It is a figure which shows an example of the reconstruction image when the resolution is gradually increased.

Hereinafter, embodiments of the present invention will be described.

FIG. 1 shows the configuration of the three-dimensional image reconstruction system 10 according to the present embodiment. As shown in FIG. 1, the three-dimensional image reconstruction system 10 includes a transmission electron microscope (TEM) 12, a drive unit 14, and an image processing device 16.

The transmission electron microscope 12 has the same configuration as the transmission electron microscope described in Patent Document 1, and is an image for detecting a projection image of an electron gun 18 that emits an electron beam E and a sample S irradiated with the electron beam E. Includes detector 20. Further, the transmission electron microscope 12 has various lens systems such as an irradiation lens system and an objective lens system, but the illustration is omitted for the sake of simplicity.

The drive unit 14 rotates a holder (not shown) that grips the sample S in the direction of arrow A with the direction along the Y axis in FIG. 1 as the tilt axis K. Thereby, the electron beam E can be irradiated to the sample S at a plurality of irradiation angles.

In the present embodiment, it is assumed that the shape of the sample S is a flat plate, and the position of the sample S shown by the solid line in FIG. 1 is the initial position. Then, it is assumed that the tilt angle of the sample S is 0 degrees when it is in the initial position of the sample S, and the tilt angle of the sample S with respect to the initial position when the sample S is rotated clockwise is defined as the tilt angle in the plus direction. The tilt angle of the sample S with respect to the initial position when the sample S is rotated counterclockwise is defined as the tilt angle in the minus direction.

In the present embodiment, for example, the sample S is tilted by a predetermined unit angle within a predetermined tilt angle range, and the sample S is photographed to obtain a plurality of projected images. The tilt angle range and the unit angle are appropriately set according to the accuracy and the like required when reconstructing the three-dimensional image of the sample S. The tilt angle range is set to a range in which the sample S can be tilted as much as possible, and is set to a range of −70 degrees to +70 degrees as an example. Further, the unit angle is set to several degrees to several tens of degrees, and is set to 10 degrees as an example. The tilt angle range and the unit angle are examples, and are not limited to these.

The image processing device 16 includes a control unit 22, an operation unit 24, a display unit 26, and a storage unit 28.

As shown in FIG. 1, the control unit 22 functionally includes an acquisition unit 30, a calculation unit 32, a generation unit 34, a rearrangement unit 36, and a repeat control unit 38.

The acquisition unit 30 acquires from the image detector 20 that acquires a plurality of first projected images at a plurality of angles with respect to the sample S. Specifically, the acquisition unit 30 acquires a plurality of first projection images obtained by irradiating the sample S with the electron beam E at a plurality of irradiation angles from the image detector 20.

The calculation unit 32 calculates the error distribution based on the plurality of second projection images at the plurality of angles of the plurality of first projection images and the reconstruction image. Specifically, the calculation unit 32 calculates the error distribution based on the plurality of second projected images when the plurality of first projected images and the reconstructed images are projected at a plurality of irradiation angles.

The generation unit 34 generates a reconstructed image by arranging the concentration quanta obtained by quantizing the concentration value based on the error distribution calculated by the calculation unit 32.

The rearrangement unit 36 rearranges the concentration quantum based on the reconstruction image and the error distribution generated by the generation unit 34. Specifically, the rearrangement unit 36 rearranges the concentration quanta by moving the concentration quanta from the pixel having the largest number of concentration quanta to the pixel having the least number of concentration quanta based on the error distribution. do.

The repetition control unit 38 controls so that the processing by the calculation unit 32, the generation unit 34, and the rearrangement unit 36 is repeated while gradually increasing the resolution of the reconstructed image. As a result, a reconstructed image of the sample S, that is, a three-dimensional image is generated.

The control unit 22 includes a CPU, a ROM, a RAM, and the like (not shown) as a hardware configuration. The CPU reads and executes the image processing program from the storage unit 28 in which the image processing program for causing the CPU to execute the image processing described later is stored.

The operation unit 24 includes a mouse, a keyboard, and the like for performing various operations.

The display unit 26 is composed of, for example, a liquid crystal display or the like.

The storage unit 28 stores the image processing program, the data of the three-dimensional image obtained by executing the image processing program, and the like.

Hereinafter, the image processing executed by the control unit 22 will be described with reference to the flowchart shown in FIG. The image processing shown in FIG. 2 is executed by reading the image processing program from the storage unit 28, for example, when the user is instructed to execute the image processing program.

In step S100, a plurality of projection images obtained by irradiating the sample S with the electron beam E at a plurality of irradiation angles are acquired. Specifically, the sample S is tilted by a predetermined unit angle within a predetermined tilt angle range, and the electron beam E is driven from the electron gun 18 with respect to the sample S at each tilt angle. The unit 14 and the electron gun 18 are controlled.

For example, if the tilt angle range is -70 degrees to 70 degrees and the unit angle is 10 degrees, then -70 degrees, -60 degrees, ..., -10 degrees, 0 degrees, 10 degrees, ..., 60 degrees, 70. Fifteen projection images taken at a degree tilt angle θ _i (i is 1 to 15) are obtained.

Then, the projected image detected by the image detector 20 at each inclination angle is acquired and stored in the storage unit 28.

In step S102, the total number N of concentration quanta (quantum unit: QUA) is calculated based on at least one projection image of the plurality of projection images acquired in step S100. Here, the density quantum is the same concept as that described in Patent Document 1 and Non-Patent Document 1, and is a unit quantum when the density value of each pixel of the three-dimensional image of the sample S is quantized. Say. Therefore, the density value of each pixel is represented by the integrated value of the density quantum. That is, one concentration quantum corresponds to the concentration value "1", and for example, when the concentration value is "50", it means that the number of concentration quanta is 50.

In the present embodiment, as an example, the value obtained by integrating the density values of each pixel of the projected image having an inclination angle of 0 degrees is defined as the total number of density quanta N. The total number of all concentration quanta of the plurality of projected images may be calculated, and the average value of the total number of the calculated total number of concentration quanta may be set to the total number N.

In step S104, the resolution r is set. Here, the resolution r means the number of pixels included in an image of a predetermined size. In the present embodiment, it is assumed that the resolution of the projected image acquired in step S100 is 512 × 512 pixels as an example. Then, in the present embodiment, the resolution r is gradually increased. Therefore, the first resolution r is set to 32 × 32 pixels as an example. Then, the resolution is gradually increased such as 64 × 64 pixels, 128 × 128 pixels, 512 × 512 pixels. As the resolution r increases stepwise, the size of the concentration quantum gradually decreases.

In step S105, it is determined whether or not the resolution r set in step S104 is the lowest resolution. In the present embodiment, as an example, it is determined whether or not the resolution r set in step S104 is 32 × 32 pixels. Then, when the resolution r in step S104 is the lowest resolution, the process proceeds to step S106. On the other hand, if the resolution r in step S104 is not the lowest resolution, that is, if the resolution r set in step S104 is higher than the lowest resolution, the process proceeds to step S107.

In step S106, the resolution r set in step S104, specifically, the lowest resolution three-dimensional reconstructed image is initialized. That is, the density values of all the pixels constituting the reconstructed image are set to "0".

In step S107, the reconstructed image calculated at the previous resolution is set as the reconstructed image necessary for calculating the error distribution calculated in step S108 described later. Specifically, the reconstructed image when the step S118 described later is executed last time, that is, the reconstructed image calculated at the previous resolution is set. More specifically, a reconstructed image obtained by converting the reconstructed image calculated at the previous resolution to the resolution r set in step S104 is set.

In step S108, the error distribution E _map (x, z) corresponding to the projected image of the tilt angle θ _i is calculated using the following equation using the same method as in Non-Patent Document 1. The index i is attached to all inclination angles θ. Then, the sum of the error distributions calculated for all the inclination angles θ is E _map .

The error distribution E _map (x, z) is a numerical value of the degree of whether or not it is necessary to place the concentration quantum at a certain coordinate (x, z) of the fault plane (XX plane). If the value is negative, it means that the pixel at that coordinate lacks concentration quantum, and if the value is positive, it means that the pixel at that coordinate has too much concentration quantum.

・・・（１）

... (1)

Here, g (x, z) represents a reconstructed image, and FP [g (x, z)] represents a projected image of the reconstructed image g (x, z). Further, p _θiexp (x') is a projection image having an inclination angle θ _i , but is a projection image obtained by performing resolution conversion according to the resolution r of the reconstructed image g (x, z). Further, BP [E _θi (x')] represents a back projection image of E _θi (x').

That is, the operation of back-projecting the difference between the projected image obtained by projecting the reconstructed image g (x, z) at the tilt angle θ _i and the projected image actually projected on the sample S at the tilt angle θ _i is performed at each tilt angle. By performing for θ _i , a three-dimensional error distribution E _map (x, y, z), which is a set of error distributions E _map (x, z) of a plurality of tomographic images along the Y axis, can be obtained.

In step S110, the total number of concentration quanta Nr is calculated based on the projected image obtained by projecting the reconstructed image having the resolution r. For example, as in step S102, the total number of density quanta Nr is the sum of the density values of each pixel of the projected image obtained by projecting the reconstructed image at an inclination angle of 0 degrees. The total number of all density quanta of the plurality of projected images obtained by projecting the reconstructed image at all the tilt angles may be calculated, and the average value of the total number of the calculated plurality of density quanta may be the total number Nr.

In step S112, the concentration quantum is reconstructed based on the error distribution E _map (x, y, z) calculated in step S108 and the total number of concentration quanta Nr of the reconstructed image of the resolution r calculated in step S110. Place it in each pixel of the image. Specifically, similarly to the method described in Non-Patent Document 1, the concentration quanta of the total number Nr are arranged in order from the pixel having the largest error according to the error distribution E _map (x, y, z).

In step S114, the concentration quantum rearrangement process is performed. Specifically, among the pixels in which the density quantum is arranged and constitutes the reconstructed image of the resolution r, for each pixel of the pixel array along the Z-axis direction of the coordinates (x, y), the error of the same coordinates of the error distribution. The pixel having the largest number of density quantums and the pixel having the smallest number of density quantums are set as a pair of pixels. That is, the first pixel having the largest positive difference between the pixel density value and the error (number of density quanta) and the second pixel having the smallest negative difference are set as paired pixels. Then, the concentration quanta are temporarily moved from the first pixel to the second pixel by a predetermined number (for example, one). FIG. 3 shows an example of setting the pixels of the pair. As shown in FIG. 3, among the pixel sequences along the arrow A direction of a certain coordinate (x, y), that is, the Z-axis direction, the number of the first pixel P1 and the density quantum having the largest number of concentration quanta is too large. The second pixel P2, which has too few, is set as a pair of pixels.

Next, the error distribution of the reconstructed image in which the concentration quantum is temporarily moved is recalculated by the above equation (1). Then, the total number of errors calculated from the error distribution before moving the concentration quantum and the total number of errors calculated from the error distribution after moving the concentration quantum are compared, and it is determined whether or not the error has decreased. do. Then, when the error is reduced, the reduced number of errors and the coordinates of the paired pixels are stored in the storage unit 28.

In this way, the process of determining the number of reductions in the error is performed for each of the pixel strings in the Z-axis direction in all combinations of coordinates (x, y). Then, the reconstructed image is updated by moving the density quantum with respect to the pair of pixels having the largest reduction in error, that is, by rearranging the density quantum.

Then, the above processes, that is, the setting of the pair of pixels, the temporary movement of the density quantum, the determination of the number of reductions in the error, and the rearrangement of the density quantum are executed until the total number of errors does not decrease. That is, the total number of errors is changed from decreasing to increasing, and the reconstruction image immediately before the increase is used as the final reconstruction image in this step.

In step S116, it is determined whether or not the processes of steps S105 to S114 have been executed for all the inclination angles θ _i , and if affirmed, the process proceeds to step S118. On the other hand, if it is denied, the process proceeds to step S105, and the processing of steps S105 to S114 is executed for the unprocessed inclination angle θ _i .

In step S118, a process of averaging the reconstructed images calculated for all the inclination angles θi is performed, and the final reconstructed image is obtained. That is, the final reconstructed image is calculated by calculating the average value of the pixels at the same coordinates of the reconstructed image calculated for all the inclination angles θ _i .

In step S120, it is determined whether or not the processing of steps S104 to S118 has been executed up to the maximum resolution, that is, 512 × 512 pixels, and if affirmed, this routine is terminated. On the other hand, if it is denied, the process proceeds to step S104, the resolution r is increased by one step, and the processes of steps S104 to S118 are executed.

FIG. 4 shows an example in which the resolution is gradually increased to generate a reconstructed image. The image G1 represents a calculation model of the sample S having a density change, and the resolution is 512 × 512 pixels. Further, the image G2 is a reconstructed image having a resolution of 32 × 32 pixels, the image G3 is a reconstructed image having a resolution of 64 × 64 pixels, and the image G4 is a reconstructed image having a resolution of 128 × 128 pixels. The image G5 is a reconstructed image having a resolution of 512 × 512 pixels. The number of gradations is 256 gradations. As shown in FIG. 4, it can be seen that the higher the resolution, the closer to the computational model.

As described above, in the present embodiment, the three-dimensional image of the sample S is not reconstructed simply by arranging the concentration quanta, but then the concentration quanta are rearranged based on the error distribution. As a result, the three-dimensional image can be reconstructed with high accuracy as compared with the case where the three-dimensional image of the sample S is reconstructed simply by arranging the concentration quanta.

In the present embodiment, the case where the sample S is rotated for shooting is described, but the irradiation direction of the electron beam E may be changed to a plurality of angles for shooting.

Further, in the present embodiment, the case where the present invention is applied to an electron beam CT (Computed Tomography) that irradiates a sample S with an electron beam E has been described, but the present invention is not limited thereto. Not limited to electron beam CT, for example, medical or industrial X-ray CT, synchrotron radiation CT, photoacoustic CT, and MRI (Magnetic Resolution Image) can be obtained to obtain a projected image of sample S to obtain a tomographic image. The present invention can be applied to any device capable of this.

Further, in the present embodiment, the case where the sample S is irradiated with the electron beam E, which is one of the particle radiations, has been described, but other particle radiations such as neutron rays, heavy particle rays, alpha rays, and beta rays are used. Alternatively, electromagnetic radiation such as X-rays and gamma rays may be used.

Further, in the present embodiment, the embodiment in which the image processing program is stored (installed) in the storage unit 28 in advance has been described, but the present invention is not limited to this. The image processing program is provided in a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versaille Disc Read Only Memory), and a USB (Universal Serial Bus) memory. good. Further, the image processing program may be downloaded from an external device via a network.

10 Three-dimensional image reconstruction system 12 Transmission electron microscope 14 Drive unit 16 Image processing device 18 Electron gun 20 Image detector 22 Control unit 24 Operation unit 26 Display unit 28 Storage unit 30 Acquisition unit 32 Calculation unit 34 Generation unit 36 Relocation Unit 38 Repeat control unit

Claims (4)

An acquisition unit that acquires a plurality of first projection images at a plurality of angles with respect to a sample,
A calculation unit that calculates an error distribution based on a plurality of second projection images at the plurality of angles of the plurality of first projection images and reconstruction images, and a calculation unit.
A generation unit that generates a reconstructed image by arranging a concentration quantum obtained by quantizing a concentration value based on the error distribution.
A rearrangement unit that rearranges the concentration quantum based on the generated reconstruction image and the error distribution, and the rearrangement unit.
Equipped with
Based on the error distribution, the rearrangement unit has the number of concentration quanta in the pixel array along the irradiation direction of the electron beam irradiated to the sample among the pixels constituting the reconstruction image. The concentration quanta are rearranged by moving the concentration quanta from the most pixels to the pixel having the smallest number of concentration quanta in the pixel array.
Image processing device. The image processing according to claim 1, further comprising a repeat control unit that controls the processing by the calculation unit, the generation unit, and the rearrangement unit while gradually increasing the resolution of the reconstructed image. Device. The computer
The step of acquiring multiple first projections at multiple angles with respect to the sample,
A step of calculating an error distribution based on a plurality of second projection images at the plurality of angles of the plurality of first projection images and the reconstruction image, and a step of calculating the error distribution.
Based on the error distribution, a step of generating a reconstructed image by arranging a concentration quantum representing a concentration unit, and
A step of rearranging the concentration quantum based on the generated reconstruction image and the error distribution, and
Including
The rearrangement step is based on the error distribution, and is the number of the concentration quanta in the pixel sequence along the irradiation direction of the electron beam irradiated to the sample among the pixels constituting the reconstructed image. The concentration quanta are rearranged by moving the concentration quanta from the pixel having the largest number to the pixel having the smallest number of concentration quanta in the pixel array.
An image processing method that performs processing. An image processing program for making a computer function as each part of the image processing apparatus according to claim 1 .

Images