KR20230055317A - System and method of converting imaginary tem sadp image to real tem sadp image or real tem sadp image to imaginary tem sadp image using deep learning - Google Patents

Abstract

A diffraction pattern image conversion system and a method of converting a virtual TEM SADP image into an actual TEM SADP image or an actual TEM SADP image into a virtual TEM SADP image using deep learning are disclosed. The diffraction pattern image conversion system comprises: an actual diffraction pattern image refining unit which removes unnecessary information from an actual diffraction pattern image; a virtual diffraction pattern generation unit which acquires a virtual diffraction pattern image corresponding to the actual diffraction pattern image with the unnecessary information removed; and an algorithm training unit which trains an algorithm capable of generating a diffraction pattern similar to the actual one using at least one of the actual diffraction pattern image, from which the unnecessary information is removed, and the virtual diffraction pattern image. Therefore, the diffraction pattern image conversion system can prevent the ringing effect of the diffraction pattern that may occur from discontinuities in the light source.

Description

Diffraction pattern image conversion system and method for mutually converting virtual TEM SADP image and real TEM SADP image using deep learning USING DEEP LEARNING}

The present invention relates to a TEM SADP image generation system and method having high discrimination by adaptively responding to input parameters.

In addition, the present invention relates to a diffraction pattern image conversion system and method for mutually converting a virtual TEM SADP image and an actual TEM SADP image using deep learning.

In the actual TEM SADP image, the electron beam is transmitted the most, so a bright spot appears in the center of the image. Since the dot has a relatively high brightness compared to surrounding diffraction dots, the surrounding diffraction dots are difficult to see. Therefore, in general, when taking a TEM SADP image, a bright spot in the center of the image is covered with a beam stopper as shown in FIG. 15. However, existing simulation programs do not consider the effect of these beam stoppers.

In addition, the actual TEM SADP has various forms, such as errors that occur when the direction of the electron beam and the positive axis do not exactly match, errors that may occur in the optical system, and errors that may occur in the process of acquiring a diffraction pattern through an image sensor such as CCD/CMOS. There is an error in However, existing simulation programs do not consider these effects.

In addition, depending on the manufacturer of the TEM, a difference in the quality of the acquired SADP image may occur or the aspect of the error may change, but the simulation program does not consider these effects.

KR 10-2021-0122161 A KR 10-2111124 B KR 10-2005508 B KR 10-1964529 B KR 10-2008-0111573 A KR 10-2221931 B KR 10-0403419 B KR 10-2021-0130953 A KR 10-1967300 B

The present invention is to provide a system and method for generating a TEM SADP image having high discriminative power by adaptively responding to input parameters.

In addition, the present invention is to provide a system and method for generating a virtual diffraction pattern image that can be used in a TEM.

In addition, the present invention is to provide a technique capable of preventing a ringing effect, a HOLZ, or a blurry diffraction point from being included in a diffraction pattern image.

Furthermore, the present invention provides a technique capable of generating a diffraction pattern image by mathematically analyzing parameters input by a user.

In addition, the present invention is to provide a computing device capable of fast processing by utilizing CPU parallel processing or GPGPU.

Moreover, the present invention provides a technique utilizing an image processing technique such as gamma correction.

Moreover, the present invention is to provide a technique for utilizing a SADP image generated in response to an input parameter adaptively.

An object of the present invention is to provide a technique capable of preventing a phenomenon in which a material is destroyed by a large amount of scanning beam output.

In addition, the present invention is to provide a diffraction pattern image conversion system and method for mutually converting a virtual TEM SADP image and a real TEM SADP image using deep learning.

In order to achieve the above object, a diffraction pattern image conversion system according to an embodiment of the present invention includes an actual diffraction pattern image refiner for removing unnecessary information from an actual diffraction pattern image; a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and generating an image belonging to the real diffraction pattern domain from an image belonging to the virtual diffraction pattern domain by using at least one of the real diffraction pattern image from which unnecessary information is removed and the virtual diffraction pattern image, or a virtual image belonging to the real diffraction pattern domain. A real-virtual mutual conversion algorithm learning unit for generating an image belonging to the diffraction pattern domain is included.

The diffraction pattern image may be a transmission electron microscope (TEM) selected area diffraction pattern (SADP) image.

The unnecessary information may be annotation, scale, or index information.

The actual diffraction pattern image refiner may detect the unnecessary information in the actual diffraction pattern image and fill in a region where the unnecessary information was present using information surrounding the unnecessary information detected using a hole-filling algorithm.

The virtual diffraction pattern generator may use a TEM SADP simulation program, and the TEM SADP simulation program may generate a virtual TEM SADP image corresponding to the actual TEM SADP image based on the input lattice constant and unit cell.

The TEM SADP simulation program may be one of JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal or Condor.

The information on the lattice constant and the unit cell may be provided in the form of a file such as CIF (Crystallography Information File), FHI-aims, or XYZ.

The virtual diffraction pattern generator may include a sample generator configured to generate a sample using at least one of a lattice constant, a relative position of an atom in a unit cell, and a positive axis parameter; a vector generator for generating a reciprocal lattice vector corresponding to the unit cell; a light source generating unit that obtains the brightness of electron beams reaching the atoms in the generated sample; a diffraction pattern generating unit generating a virtual diffraction pattern image using the generated reciprocal lattice vector, the position of an atom in the sample, and the obtained brightness of the electron beam; and a selection unit that selects a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images.

The sample generation unit adjusts the number of layers of the slab according to the lattice constant and positive axis parameter among the input parameters to prevent a phenomenon in which a high order Laue Zone (HOLZ) is included in a diffraction pattern or a diffraction pattern with blurry diffraction points is generated In order to prevent the ringing effect of the diffraction pattern that may occur from the discontinuous point of the light source, the light source generator adjusts the shape of the light source and the intensity of the light source according to the input size of the slab or the size of the diffraction pattern image. can be changed to

The real-virtual mutual conversion algorithm learning unit may generate a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning model.

The real-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to the virtual diffraction pattern domain using a specific algorithm when the real diffraction pattern image and the virtual diffraction pattern image are prepared. can do.

The real-virtual mutual conversion algorithm learning unit Real2Sim conversion unit; Sim2Real conversion unit; an actual diffraction pattern classification unit for learning a deep learning model that distinguishes between a virtual diffraction pattern converted into a form similar to the actual diffraction pattern through the Sim2Real conversion unit and a diffraction pattern photographed through an actual TEM; and a virtual diffraction pattern discriminator for learning a deep learning model that distinguishes between an actual diffraction pattern converted into a form similar to the virtual diffraction pattern through the Real2Sim converter and a virtual diffraction pattern generated through simulation, wherein the Real2Sim converter includes the real diffraction pattern. A deep learning model for converting an image belonging to the diffraction pattern domain into an image belonging to the virtual diffraction pattern domain is trained, and the Sim2Real conversion unit trains a deep learning model for converting an image belonging to the virtual diffraction pattern domain into an image belonging to the actual diffraction pattern domain. can be learned.

A diffraction pattern image conversion system according to another embodiment of the present invention includes an actual diffraction pattern image refiner for removing unnecessary information from an actual diffraction pattern image; a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and learning an algorithm for generating a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning algorithm learned using at least one of the actual diffraction pattern image and the virtual diffraction pattern image. includes wealth

A diffraction pattern image conversion system according to another embodiment of the present invention includes an actual diffraction pattern image refiner for removing unnecessary information from an actual diffraction pattern image; a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and learning an algorithm for generating a diffraction pattern image belonging to a virtual diffraction pattern domain from a diffraction pattern image belonging to an actual diffraction pattern domain by using a deep learning algorithm learned using at least one of the actual diffraction pattern image and the virtual diffraction pattern image. includes wealth

A diffraction pattern image conversion system according to another embodiment of the present invention includes a virtual diffraction pattern generator for acquiring a virtual diffraction pattern image corresponding to an actual diffraction pattern image; and an algorithm learning unit for learning an algorithm capable of generating a diffraction pattern similar to a real one from a virtual diffraction pattern using at least one of the real diffraction pattern image and the virtual diffraction pattern image. The virtual diffraction pattern generating unit may include a sample generating unit generating a slab-type sample using parameters input by a user; a vector generator that generates a reciprocal lattice vector that meets the Eward sphere by using image coordinates separated by a predetermined distance from the origin where the electron beam is located and the wavelength of the electron beam; a light source generating unit that obtains the brightness of an electron beam reaching each atom in the sample by using the input shape and intensity of the light source; An accumulated diffraction pattern is calculated using the generated reciprocal lattice vector, the position of an atom in the sample, and the obtained brightness of the electron beam, a maximum value of the accumulated diffraction pattern is calculated, and based on the calculated maximum value, the accumulated diffraction pattern is calculated. a diffraction pattern generating unit generating a diffraction pattern image by linearly normalizing the accumulated diffraction patterns; and a selector for selecting a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images. Here, the diffraction pattern image is a TEM SADP image, and the sample generator generates a lattice constant and The number of layers of the slab is adaptively determined according to the positive axis parameter, and the light source generation unit determines the shape of the light source and the intensity of the light source to prevent the ringing effect of the diffraction pattern that may occur from the discontinuous point of the light source. It is adaptively varied according to the size of or the size of the diffraction pattern image to be generated.

A diffraction pattern image conversion method according to an embodiment of the present invention includes removing unnecessary information from an actual diffraction pattern image; generating a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information is removed; and generating an image belonging to the real diffraction pattern domain from an image belonging to the virtual diffraction pattern domain by using at least one of the real diffraction pattern image from which unnecessary information is removed and the virtual diffraction pattern image, or a virtual image belonging to the real diffraction pattern domain. and generating an image belonging to the diffraction pattern domain.

The TEM SADP image generation system and method according to the present invention adaptively responds to input parameters to prevent a phenomenon in which a High Order Laue Zone (HOLZ) is included in a diffraction pattern or a diffraction pattern with blurred diffraction points is generated, and discontinuous points of a light source It is possible to prevent the ringing effect of the diffraction pattern that may occur from

In addition, the diffraction pattern image conversion system of the present invention generates a TEM SADP image similar to a real one corresponding to a virtual TEM SADP image, and thus can simulate various errors that may occur in an actual TEM experiment.

1 is a diagram showing an example of an actual TEM SADP image.
2 is a diagram illustrating a virtual SADP image generated by a JEMS program.
3 is a diagram illustrating a SADP image including a HOLZ pattern.
4 is a diagram illustrating a SADP image including a blurred diffraction point pattern.
5 is a diagram illustrating a SADP image including a ringing effect.
6 is a block diagram schematically showing the configuration of a TEM SADP image generating system according to an embodiment of the present invention.
7 is a diagram illustrating a SADP image in which the ringing effect has disappeared, generated using the TEM SADP image generation method of the present invention.
8 is a diagram showing an example of lattice constants of a material belonging to a cubic system.
9 is a diagram showing an example of lattice constants of a material belonging to a hexagonal system.
10 is a diagram showing the relationship between an electron beam, an Eward sphere, a reciprocal lattice, and a diffraction pattern.
11 is a diagram showing before and after aligning unit cells.
12 is a diagram showing the result of making a slab-shaped sample using a unit cell aligned with the positive axis according to an embodiment of the present invention.
13 is a diagram showing an example of a diffraction pattern generated by the TEM SADP image generating system of the present invention.
14 is a diagram illustrating a system for mutually converting a real TEM SADP image and a virtual TEM SADP image according to an embodiment of the present invention.
15 is a diagram illustrating an example of blocking a bright point in the center of an image using a beam stopper.
16 is a diagram showing an example of a real TEM SADP image with scale information displayed.
17 is a diagram showing an example of deleting scale information from an actual TEM SADP image.
18 is a diagram showing an example of a virtual TEM SADP image generated through a JEMS program.
19 is a diagram illustrating an example of a TEM SADP image generated from a virtual TEM SADP image in a form similar to a real one.
20 is a diagram showing an example of a real-virtual mutual conversion algorithm according to an embodiment of the present invention.

Singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some of the steps It should be construed that it may not be included, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

The present invention relates to a system and method for generating a TEM (Transmission Electron Microscope) SADP (Selected Area Diffraction Pattern) image having high discrimination ability by adaptively responding to an input parameter, The ringing effect of the diffraction pattern of the rotation pattern that can occur from TEM SADP images can be created.

In order to grasp the characteristics of a material, an electron beam is scanned through a material through a TEM to acquire a SADP image, but the material may be destroyed if the number of times the electron beam is output increases. Therefore, the present invention can provide SADP images with a program without actually scanning electron beams to prevent the material from being destroyed. Accordingly, the SADP image generated in this way can be used in various fields.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

6 is a block diagram schematically showing the configuration of a TEM SADP image generating system according to an embodiment of the present invention, and FIG. It is an illustrated drawing.

Referring to FIG. 6, the TEM SADP image generation system according to the present embodiment adaptively responds to the input parameters to generate a SADP image in which the ringing effect of the diffraction pattern, the phenomenon included in the HOLZ diffraction pattern, and the phenomenon including blurry diffraction points do not occur. can create

This TEM SADP image generating system generally includes a parameter setting unit 600, a sample generator 602, an HKL vector generator 604, a light source generator 606, a diffraction pattern generator 608, and their operations. It may include a control unit (not shown) for controlling. Here, the TEM SADP image generating system may be one device, for example, a server, or may be collectively referred to as a computing device.

The parameter setting unit 600 may set parameters for SADP image generation. For example, the parameter setting unit 600 may set a parameter by receiving a user's input.

According to an embodiment, the parameter setting unit 600 is configured to determine a lattice constant, a relative position of an atom in a unit cell, a zone axis, a wavelength and intensity of an electron beam, a camera distance, a diffraction pattern image size, and the like. Parameters can be set. All of these parameters may be input by the user, and other parameters may be automatically generated when the user partially inputs them.

The sample generator 602 may generate a slab-shaped sample using the relative positions of atoms in the unit cell and the positive axis parameters. Here, the slab shape may mean a thin plate shape. Of course, the resulting sample is not limited to the slab type.

The HKL vector generator 604 may generate a reciprocal lattice vector that meets the virtual Eward sphere. Here, the reciprocal lattice may be a parameter automatically generated by using a specific program according to the unit cell set by the parameter setting unit 600 .

The light source generator 606 may calculate relative brightness of electron beams reaching each atom in the sample.

The diffraction pattern generation unit 608 may generate a virtual SADP image by accumulating diffraction generated from interactions between all atoms and electrons included in the sample. In this process, the set parameters, the reciprocal lattice vector, and the relative brightness of electron beams reaching atoms may be used.

In summary, the TEM SADP image generation system of the present embodiment adaptively generates a virtual SADP image in response to various input parameters, but the SADP image has a ringing effect, a phenomenon in which a HOLZ pattern is included in a diffraction pattern, and a blurry diffraction point diffraction A phenomenon included in the pattern may not occur.

In addition, the TEM SADP image generation system can generate SADP images at high speed by utilizing parallel processing or GPGPU.

That is, the TEM SADP image generating system can generate a large amount of virtual SADP images at high speed, and the generated SADP images may be substantially the same as actual SADP images.

Meanwhile, although parameters input by the user have been specifically mentioned above, the parameters are not limited as long as a sample is generated using the parameters input by the user.

That is, the TEM SADP image generating system includes a sample generator for generating a slab-type sample using parameters input by a user, and an electron beam reaching atoms in the sample using the input shape and intensity of the light source. It may include a light source generator for obtaining brightness and a diffraction pattern generator for generating a virtual diffraction pattern image using the atomic positions in the sample and the obtained brightness of the electron beam.

Meanwhile, the reciprocal lattice vector, the brightness of the electron beam, and the diffraction pattern may be generated by mathematically applying parameters input by the user. A detailed explanation of this will be given later.

In addition, although a user inputs parameters to generate a TEM SADP image, parameters may be extracted from an actual SADP image and a virtual TEM SADP image may be generated using the extracted parameters. That is, the TEM SADP image generating system may generate a plurality of virtual TEM SADP images based on actual SADP images.

In addition, the number of layers of the slab, the reciprocal lattice vector, and the brightness of the electron beam are not fixed, and may be adaptively changed according to parameters input by the user or parameters extracted from the actual SADP image. A detailed explanation of this will be given later.

Hereinafter, a process of generating a TEM SADP image will be described in detail with reference to the accompanying drawings.

8 is a diagram showing an example of lattice constants of a material belonging to a cubic system. 9 is a diagram showing an example of the lattice constant of a material belonging to the hexagonal system, FIG. 10 is a diagram showing the relationship between an electron beam, an Eward sphere, a reciprocal lattice, and a diffraction pattern, and FIG. 11 is a unit It is a diagram showing before and after aligning the grid. 12 is a view showing the result of making a slab-shaped sample using a unit cell aligned with the positive axis according to an embodiment of the present invention, and FIG. 13 is a diagram showing the diffraction generated by the TEM SADP image generating system of the present invention It is a drawing showing an example of a pattern.

The parameter setting unit 600 may set parameters such as a lattice constant, a relative position of an atom in a unit lattice, a positive axis, a wavelength and intensity of an electron beam, a camera distance, and a size of a diffraction pattern image. These parameters may be input by the user or extracted from actual SADP images.

In this case, the lattice constant and the relative position of the atoms in the unit cell may be input in the form of a file such as CIF (Crystallography Information File), FHI-aims, or XYZ.

The sample generating unit 602 may generate a slab-shaped sample using input lattice constants, relative positions of atoms in the unit cell, and parameters for the positive axis.

Specifically, the lattice constant may be composed of six variables of magnitudes a, b, and c of lattice vectors and angles α, β, and γ between lattice vectors. As shown in FIG. 8, if a=b=c, α=β=γ=90°, the material belongs to the cubic system, and as shown in FIG. 9, if a=b≠c, α=β=90°, γ=120°, The substance belongs to the hexagonal system.

The relative positions of atoms in the unit cell can be represented as shown in Table 1 below when the three-dimensional space in the unit cell is expressed as between 0 and 1.

(The relative position of atoms in the unit cell of LiAl material belonging to the cubic system) atomic symbol relative position x-axis y axis z axis Li 0.0000 0.0000 0.5000 Al 0.0000 0.0000 0.0000

As shown in FIG. 11 , the sample generating unit 602 may generate a slab-shaped sample as shown in FIG. 12 by aligning the unit cell so that the direction of the electron beam and the lattice plane corresponding to the positive axis are perpendicular to each other. Here, the sample in the form of a slab may mean a structure in which unit cells are arranged in a plate shape. In FIG. 12, a cubic material was produced, but a hexagonal material may also be produced in the same manner. At this time, the Rodrigues formula can be used to align the direction of the electron beam and the positive axis, which can be expressed as a 3-dimensional vector.

According to an embodiment, it is possible to adaptively determine the number of layers of a slab according to the input lattice constant and the positive axis parameter to prevent the generation of a diffraction pattern in which the HOLZ is included in the diffraction pattern or the diffraction point is blurred.

That is, the sample generator 602 can adaptively determine the number of layers of the slab according to the input lattice constant and positive axis parameter to prevent the generation of a diffraction pattern in which the HOLZ is included in the diffraction pattern or the diffraction point is blurred. there is. Unit lattices may be aligned on the layers of the slab determined in this way. As a result, the number of layers of a slab may vary according to parameters input by the user, even if the same material is used.

As shown in FIG. 10, the HKL vector generator 604 may generate a reciprocal lattice vector that meets the virtual Eward sphere. Diffraction may occur at a reciprocal lattice meeting the Eward sphere, and thus, a reciprocal lattice where diffraction occurs may be detected in order to obtain a diffraction pattern. Here, the reciprocal lattice may be a parameter automatically generated by using a specific program or equations according to the unit cell set by the parameter setting unit 600 .

Specifically, the HKL vector generator 604 uses the image coordinates (x,y) separated by a predetermined distance (d) from the origin where the electron beam is located and the wavelength (λ) of the electron beam to use the reciprocal lattice vector h(x,y). ), k (x, y), l (x, y) can be calculated by Equations 1 and 2 below.

As shown in Equation 1, χ can be obtained if the image coordinates (x,y), the wavelength (λ), and the distance (d) from the origin where the electron beam is located are known. Using the obtained χ, the reciprocal lattice vector [h (x,y), k(x,y), l(x,y)] can be automatically obtained.

The light source generator 606 may obtain the brightness of the electron beam reaching each atom in the sample by receiving the shape and intensity of the light source as inputs. At this time, the shape of the light source may be flat or may have a 2D Gaussian shape based on a lattice plane perpendicular to the direction of the electron beam.

When the three-dimensional position of an atom in a sample is (x _j , y _j , z _j ), assuming the shape of a flat light source, the brightness of an electron beam reaching each atom is as shown in Equation 3 below.

는 광원의 세기를 나타낸다. here,

represents the intensity of the light source.

Assuming the form of a 2D Gaussian light source, the brightness of the electron beam reaching each atom is expressed in Equation 4 below.

는 x축으로의 표준편차를 의미하며,

는 y축으로의 표준편차를 나타낸다. here,

is the standard deviation along the x-axis,

represents the standard deviation along the y-axis.

Assuming the shape of a 3D Gaussian light source, the brightness of an electron beam reaching each atom is expressed in Equation 5 below.

는 z축으로의 표준편차를 의미한다. here,

is the standard deviation along the z-axis.

The light source generating unit 606 may utilize a 3D Gaussian to create a continuous light source shape. At this time, 3σ of the Gaussian can be set to be smaller than the width, length, and height of the sample so that a discontinuity does not occur at the edge of the sample.

According to another embodiment, the light source generating unit 606 simultaneously utilizes the 2D Gaussian and the exponential decay function to remove discontinuities that may occur in the horizontal and vertical directions of the sample with the 2D Gaussian, and the exponential decay function with the sample. A method of removing discontinuities that may occur in the height direction of can be used.

On the other hand, by applying exponential decay based on the direction of the electron beam, it is possible to simulate the falling brightness as the electron beam passes through the sample. The light source to which exponential decay is applied can be defined as in Equation 6 below.

는 exponential decay의 파라미터를 나타낸다. here,

represents the parameter of exponential decay.

The size and shape of the light source generated by the light source generator 606 may be adaptively varied according to the size of the input slab and the size of the diffraction pattern image, and as a result, diffraction that may occur from the discontinuity of the light source It is possible to prevent the ringing effect of the pattern, which is shown in FIG. 7 . That is, the light source generator 606 calculates the size and shape of the light source as the input size of the slab and the diffraction pattern image in order to prevent the ringing effect of the diffraction pattern. It can be used by adaptively varying it according to the size of .

)를 이용하여 하기 수학식 7과 같이 누적된 회절 패턴(F(h,k,l))을 계산할 수 있다. The diffraction pattern generator 608 uses the reciprocal lattice vectors [h(x,y), k(x,y), l(x,y)] obtained by the HKL vector generator 604, the positions of atoms in the sample and the light source. The brightness of the electron beam obtained by the generator 606 (

) can be used to calculate the accumulated diffraction pattern F(h,k,l) as shown in Equation 7 below.

은 j번째 원자의 산란 인자(scattering factor)를 의미한다. 이 산란 인자는 원자의 종류에 따라 다를 수 있다. here,

Means the scattering factor of the j-th atom. This scattering factor may vary depending on the type of atom.

Subsequently, the diffraction pattern generation unit 608 may generate a diffraction pattern image by calculating a maximum value of the accumulated diffraction patterns and linearly normalizing the accumulated diffraction patterns based on the calculated maximum value.

According to another embodiment, the diffraction pattern generation unit 608 may generate a diffraction pattern image by performing nonlinear normalization using an image processing technique such as gamma correction. At this time, in the diffraction pattern generator 608, since the diffraction generated from the interaction of electrons and each atom included in the sample can be independently calculated, CPU parallel processing or GPGPU (General Purpose computing on Graphics Processing Unit) is used. It can be used to calculate quickly. The SADP image thus generated is shown in FIG. 13 . As shown in FIG. 13, a ringing effect, a phenomenon in which HOLZ is included in a diffraction pattern, and a diffraction pattern in which diffraction points are blurred do not occur in the SADP image.

와 같이 감마 보정에서 사용되는 함수를 사용하여 회절 패턴을 생성할 수 있다. Meanwhile, the diffraction pattern generation unit 608 may apply various functions to generate the accumulated diffraction values into a SADP image. For example, the diffraction pattern generation unit 608 may generate a diffraction pattern by applying a linear function,

A diffraction pattern can be generated using a function used in gamma correction, such as

In summary, the SADP image generation system of this embodiment uses the reciprocal lattice vector, the position of atoms in the sample, and the brightness of the electron beam to prevent the ringing effect, the phenomenon in which the HOLZ is included in the diffraction pattern, and the phenomenon in which a diffraction pattern with blurred diffraction points is generated. SADP images can be created.

Hereinafter, a system and method for converting a real TEM SADP image and a virtual TEM SADP image using deep learning will be described with reference to the accompanying drawings.

14 is a diagram showing a system for mutually converting a real TEM SADP image and a virtual TEM SADP image according to an embodiment of the present invention, and FIG. 15 shows an example of blocking a bright point in the center of an image using a beam stopper. FIG. 16 is a diagram showing an example of an actual TEM SADP image with scale information displayed. 17 is a diagram showing an example of erasing scale information from a real TEM SADP image, FIG. 18 is a diagram showing an example of a virtual TEM SADP image generated through a JEMS program, and FIG. It is a diagram showing an example of a TEM SADP image generated in a similar form. 20 is a diagram showing an example of a real-virtual mutual conversion algorithm according to an embodiment of the present invention.

The system for converting the real TEM SADP image and the virtual TEM SADP image of the present embodiment includes a real diffraction pattern image refiner 1400, a virtual diffraction pattern generator 1402, a real-virtual mutual conversion algorithm learning unit 1404, and Includes a control unit for overall control of the operation of.

The actual diffraction pattern image refining unit 1400 may remove unnecessary parts from an actual TEM SADP image, eg, an actual TEM SADP image collected through an experiment or an actual TEM SADP image collected through the web.

For example, as shown in FIG. 15 , the actual diffraction pattern image refiner 1400 may remove added annotation information, scale information, lattice plane index information, etc. to write additional information in the actual TEM SADP image.

According to an embodiment, the actual diffraction pattern image refining unit 1400 erases annotation information from the actual TEM SADP image and uses a commercial image processing program such as Photoshop or a hole-filling algorithm to fill in the lost part of the image. background can be synthesized. A SADP image obtained through this process is shown in FIG. 17 . Of course, there is no limit to the program as long as it fills in the erased part.

As another example, the actual diffraction pattern image refiner 1400 deletes scale information from the actual TEM SADP image on which scale information is displayed as shown in FIG. A SADP image as shown can be generated.

The virtual diffraction pattern generator 1402 generates a virtual diffraction pattern corresponding to the real TEM SADP image using a TEM SADP simulation program such as JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal, and Condor based on the input lattice constant and unit cell. TEM SADP images can be created. Here, the lattice constant and the unit lattice may be input by a user or obtained by extracting from the actual TEM SADP image.

When information on the lattice constant and unit cell is given in the form of a file such as CIF (Crystallography Information File), FHI-aims, or XYZ, a virtual TEM SADP image generated using the JEMS program is shown in FIG. 18 . At this time, since the generated virtual SADP image is used for learning the real-virtual mutual conversion algorithm later, it is better to generate the SADP image using one program than to generate the SADP image using various programs in the learning process. Confusion caused by the diversity of data can be reduced.

For example, virtual TEM SADP images corresponding to real TEM SADP images from which unnecessary parts have been removed may be generated using the same JEMS program, and other programs may not be used to generate virtual TEM SADP images.

According to another embodiment, the virtual diffraction pattern generation unit 1402 may select a virtual SADP image corresponding to the real TEM SADP image from among the virtual TEM SADP images generated in FIGS. 6 to 13 . In another aspect, the virtual diffraction pattern generation unit 1402 may generate a virtual SADP image corresponding to the actual TEM SADP image through the method described in FIGS. 6 to 13 .

In this case, the virtual diffraction pattern generator 1402 generates a sample using at least one of a lattice constant, a relative position of an atom in a unit cell, and a positive axis parameter, and a reciprocal lattice vector corresponding to the unit cell. A vector generator to generate a virtual diffraction pattern image using a light source generator to obtain the brightness of an electron beam reaching atoms in the sample, the generated reciprocal lattice vector, the position of an atom in the sample, and the obtained brightness of the electron beam. It may include a diffraction pattern generation unit that generates (TEM SADP image) and a selection unit that selects a virtual diffraction pattern image corresponding to the actual diffraction pattern image among the generated virtual diffraction pattern images.

The real-virtual mutual conversion algorithm learning unit 1404 utilizes the real SADP image and the virtual SADP image to generate an image belonging to the real diffraction pattern domain from an image belonging to the virtual diffraction pattern domain or to belong to the real diffraction pattern domain. A real-virtual mutual conversion algorithm for generating an image belonging to a virtual diffraction pattern domain from an image may be trained.

Subsequently, the real-virtual mutual conversion algorithm learning unit 1404 utilizes a deep learning model, that is, by using deep learning technology, the SADP image belonging to the virtual diffraction pattern domain can be generated from the SADP image belonging to the real diffraction pattern domain. . For example, as shown in FIG. 19 , a SADP image generated in a form similar to a real one shown on the right may be generated from a virtual TEM SADP image shown on the left.

At this time, if a virtual diffraction pattern having the same diffraction point position as the actual diffraction pattern is prepared, an algorithm requiring image pairs belonging to two domains may be used. On the other hand, when only one of the actual rotation pattern and the virtual diffraction pattern is prepared, an algorithm that does not require information on image pairs belonging to two domains can be used.

Through this series of processes, the real-virtual mutual conversion algorithm can generate a TEM SADP image similar to the real one by comprehensively considering the effects of various errors included in the actual TEM SADP image and the effect of the beam stopper.

According to another embodiment, the real-virtual mutual conversion algorithm learning unit 1404 can generate a SADP image belonging to a real diffraction pattern domain from a SADP image belonging to a virtual diffraction pattern domain, and also a SADP belonging to an actual diffraction pattern domain. A SADP image belonging to a virtual diffraction pattern domain may be generated from the image. If you create a deep learning application based on TEM SADP images, the input will be actual TEM SADP images.

At this time, if the input SADP image belonging to the actual diffraction pattern domain is converted into a SADP image belonging to the virtual diffraction pattern domain, the deep learning model learned with the SADP image belonging to the virtual diffraction pattern domain can be used.

Looking at the real-virtual mutual conversion algorithm learning unit 1404 in detail with reference to FIG. 20, the real-virtual mutual conversion algorithm learning unit 1404 includes a real diffraction pattern discrimination unit; a virtual diffraction pattern classification unit; Real2Sim conversion unit; A Sim2Real conversion unit may be included.

The real diffraction pattern classification unit learns a deep learning model that distinguishes between a virtual diffraction pattern converted into a form similar to the actual diffraction pattern through the Sim2Real conversion unit and a diffraction pattern captured through the actual TEM. At this time, both the virtual diffraction pattern converted into a form similar to the actual diffraction pattern and the diffraction pattern photographed through the actual TEM are images belonging to the real diffraction pattern domain.

The virtual diffraction pattern discrimination unit learns a deep learning model that distinguishes between an actual diffraction pattern converted into a form similar to the virtual diffraction pattern through the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation. At this time, both the actual diffraction pattern converted into a form similar to the virtual diffraction pattern and the virtual diffraction pattern generated through simulation are images belonging to the virtual diffraction pattern domain.

The Real2Sim conversion unit learns a deep learning model that converts an image belonging to an actual diffraction pattern domain into an image belonging to a virtual diffraction pattern domain. At this time, the goal of the Sim2Real conversion unit is to generate an image that is so similar that the virtual diffraction pattern classifier cannot distinguish between an image generated by itself and an image generated through simulation.

The Sim2Real conversion unit learns a deep learning model that converts an image belonging to a virtual diffraction pattern domain into an image belonging to an actual diffraction pattern domain. At this time, the goal of the Real2Sim conversion unit is to generate an image so similar that the actual diffraction pattern classification unit cannot distinguish between the image it generated and the image captured through the actual TEM.

The four components belonging to the real-virtual mutual conversion algorithm learning unit 1404 influence each other and learn a deep learning model that can perform well for the given goal, and finally the actual diffraction pattern. Images belonging to the domain and images belonging to the virtual diffraction pattern domain can be seamlessly converted to each other.

Since the process of acquiring actual TEM SADP images through experiments requires a lot of time, if a deep learning model trained with SADP images belonging to the virtual diffraction pattern domain can be used in real applications, a large training dataset can be easily obtained. Because of this, it is possible to build applications with high performance.

In summary, the system (image conversion system) for converting between real TEM SADP images and virtual TEM SADP images of this embodiment removes unnecessary information from the collected real TEM SADP images, and corresponds to the real TEM SADP images from which unnecessary information has been removed It is possible to generate a virtual TEM SADP image to be generated, and to learn a real-virtual mutual conversion algorithm by utilizing the generated virtual TEM SADP image and the real TEM SADP image. At this time, the real-virtual mutual conversion algorithm may generate a SADP image belonging to the real diffraction pattern domain from an SADP image belonging to the virtual diffraction pattern domain, or a SADP image belonging to the virtual diffraction pattern domain from the SADP image belonging to the real diffraction pattern domain. can also create

By generating a TEM SADP image similar to the real one from the virtual TEM SADP, various errors that may occur in an actual TEM experiment can be simulated.

In addition, when creating a deep learning application using TEM SADP images, virtual data generated by simulation can be effectively used for deep learning learning by reducing the gap between the real TEM SADP and the virtual TEM SADP through a real-virtual mutual conversion algorithm. there is.

On the other hand, the components of the above-described embodiment can be easily grasped from a process point of view. That is, each component can be identified as each process. In addition, the process of the above-described embodiment can be easily grasped from the viewpoint of components of the device.

In addition, the technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiments or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. A hardware device may be configured to act as one or more software modules to perform the operations of the embodiments and vice versa.

The embodiments of the present invention described above have been disclosed for illustrative purposes, and those skilled in the art having ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions will be considered to fall within the scope of the following claims.

600: parameter setting unit 602: sample generating unit
604: HKL vector generator 606: light source generator
608: diffraction pattern generation unit 1400: actual diffraction pattern image refinement unit
1402: virtual diffraction pattern generation unit 1404: real-virtual mutual conversion algorithm learning unit

Claims (20)

an actual diffraction pattern image refiner that removes unnecessary information from the actual diffraction pattern image;
a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and
An image belonging to the real diffraction pattern domain is generated from an image belonging to the virtual diffraction pattern domain by using at least one of the real diffraction pattern image from which the unnecessary information is removed and the virtual diffraction pattern image, or a virtual diffraction pattern image belonging to the real diffraction pattern domain is used. A diffraction pattern image conversion system comprising a real-virtual mutual conversion algorithm learning unit for generating an image belonging to a pattern domain. The diffraction pattern image conversion system according to claim 1, wherein the diffraction pattern image is a TEM (Transmission Electron Microscope) SADP (Selected Area Diffraction Pattern) image. The image conversion system according to claim 2, wherein the unnecessary information is information on annotation, scale, or index. 4. The method of claim 3, wherein the actual diffraction pattern image refiner detects the unnecessary information in the actual diffraction pattern image and fills an area where the unnecessary information exists using information surrounding the unnecessary information detected using a hole-filling algorithm. Diffraction pattern image conversion system, characterized in that the dreaming. The method of claim 2, wherein the virtual diffraction pattern generator uses a TEM SADP simulation program,
The TEM SADP simulation program generates a virtual TEM SADP image corresponding to the actual TEM SADP image based on the input lattice constant and unit cell. The diffraction pattern image conversion system according to claim 5, wherein the TEM SADP simulation program is one of JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal, and Condor. The diffraction pattern image conversion system according to claim 6, wherein the information on the lattice constant and the unit lattice can be provided in a file format such as CIF (Crystallography Information File), FHI-aims, or XYZ. The method of claim 2, wherein the virtual diffraction pattern generator,
a sample generator for generating a sample using at least one of a lattice constant, a relative position of an atom in a unit cell, and a positive axis parameter;
a vector generator for generating a reciprocal lattice vector corresponding to the unit cell;
a light source generating unit that obtains the brightness of electron beams reaching the atoms in the generated sample;
a diffraction pattern generating unit generating a virtual diffraction pattern image using the generated reciprocal lattice vector, the position of an atom in the sample, and the obtained brightness of the electron beam; and
And a selection unit for selecting a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images. 9. The method of claim 8 , wherein the sample generating unit prevents a phenomenon in which a high order laue zone (HOLZ) is included in a diffraction pattern or a diffraction pattern in which diffraction points are blurred is generated according to a lattice constant and a positive axis parameter among the input parameters. slab Adaptively determines the number of layers of
The light source generating unit adaptively changes the shape of the light source and the intensity of the light source according to the size of the input slab or the size of the diffraction pattern image to prevent the ringing effect of the diffraction pattern that may occur from the discontinuous point of the light source. Characterized by a diffraction pattern image conversion system. The diffraction pattern image of claim 2 , wherein the real-virtual mutual conversion algorithm learning unit generates a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning model. conversion system. 11. The method of claim 10, wherein the real-virtual mutual conversion algorithm learning unit converts a diffraction pattern image belonging to the virtual diffraction pattern domain to a real diffraction pattern domain by using a specific algorithm when the real diffraction pattern image and the virtual diffraction pattern image are prepared. A diffraction pattern image conversion system, characterized in that for generating a diffraction pattern image belonging to. The method of claim 2, wherein the real-virtual mutual conversion algorithm learning unit,
Real2Sim conversion unit;
Sim2Real conversion unit;
an actual diffraction pattern classification unit for learning a deep learning model that distinguishes between a virtual diffraction pattern converted into a form similar to the actual diffraction pattern through the Sim2Real conversion unit and a diffraction pattern photographed through an actual TEM; and
A virtual diffraction pattern classification unit for learning a deep learning model that distinguishes between an actual diffraction pattern converted into a form similar to the virtual diffraction pattern through the Real2Sim conversion unit and a virtual diffraction pattern generated through simulation,
The Real2Sim converter trains a deep learning model that converts an image belonging to the real diffraction pattern domain into an image belonging to the virtual diffraction pattern domain,
The diffraction pattern image conversion system, characterized in that the Sim2Real conversion unit learns a deep learning model that converts an image belonging to a virtual diffraction pattern domain to an image belonging to an actual diffraction pattern domain. an actual diffraction pattern image refiner that removes unnecessary information from the actual diffraction pattern image;
a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and
An algorithm learning unit for generating a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to a virtual diffraction pattern domain by using a deep learning algorithm learned using at least one of the actual diffraction pattern image and the virtual diffraction pattern image. Diffraction pattern image conversion system, characterized in that it comprises. The method of claim 13 , wherein the algorithm learning unit converts a diffraction pattern image belonging to an actual diffraction pattern domain from a diffraction pattern image belonging to the virtual diffraction pattern domain using a specific algorithm when the actual diffraction pattern image and the virtual diffraction pattern image are prepared. Diffraction pattern image conversion system, characterized in that for generating. an actual diffraction pattern image refiner that removes unnecessary information from the actual diffraction pattern image;
a virtual diffraction pattern generator that obtains a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information has been removed; and
An algorithm learning unit for generating a diffraction pattern image belonging to a virtual diffraction pattern domain from a diffraction pattern image belonging to an actual diffraction pattern domain by using a deep learning algorithm learned using at least one of the actual diffraction pattern image and the virtual diffraction pattern image. Diffraction pattern image conversion system, characterized in that it comprises. a virtual diffraction pattern generator for acquiring a virtual diffraction pattern image corresponding to the actual diffraction pattern image; and
An algorithm learning unit for learning an algorithm capable of generating a diffraction pattern similar to a real one from a virtual diffraction pattern using at least one of the real diffraction pattern image and the virtual diffraction pattern image,
The virtual diffraction pattern generator,
a sample generating unit generating a slab-type sample using parameters input by a user;
a vector generator that generates a reciprocal lattice vector that meets the Eward sphere by using image coordinates separated by a predetermined distance from the origin where the electron beam is located and the wavelength of the electron beam;
a light source generating unit that obtains brightness of an electron beam reaching each atom in the sample by using the type and intensity of the light source input;
An accumulated diffraction pattern is calculated using the generated reciprocal lattice vector, the position of an atom in the sample, and the obtained brightness of the electron beam, a maximum value of the accumulated diffraction pattern is calculated, and the a diffraction pattern generating unit generating a diffraction pattern image by linearly normalizing the accumulated diffraction patterns; and
A selection unit for selecting a virtual diffraction pattern image corresponding to the actual diffraction pattern image from among the generated virtual diffraction pattern images,
The diffraction pattern image is a TEM SADP image,
The sample generation unit adjusts the number of layers of the slab according to the lattice constant and the positive axis parameter among the input parameters to prevent a phenomenon in which the HOLZ (High Order Laue Zone) is included in the diffraction pattern or a diffraction pattern with blurry diffraction points is generated determined by
The light source generation unit adaptively varies the shape of the light source and the intensity of the light source according to the size of the input slab or the size of the diffraction pattern image to be generated to prevent the ringing effect of the diffraction pattern that may occur from the discontinuous point of the light source Diffraction pattern image conversion system, characterized in that to do. removing unnecessary information from the actual diffraction pattern image;
generating a virtual diffraction pattern image corresponding to the actual diffraction pattern image from which the unnecessary information is removed; and
An image belonging to the real diffraction pattern domain is generated from an image belonging to the virtual diffraction pattern domain by using at least one of the real diffraction pattern image from which the unnecessary information is removed and the virtual diffraction pattern image, or a virtual diffraction pattern image belonging to the real diffraction pattern domain is used. A diffraction pattern image conversion method comprising the step of generating an image belonging to a pattern domain. 18. The diffraction pattern of claim 17, wherein the unnecessary information is detected from the actual diffraction pattern image, and the area where the unnecessary information is located is filled using information surrounding the unnecessary information detected using a hole-filling algorithm. How to convert video. The method of claim 18, wherein a virtual diffraction pattern image corresponding to the actual diffraction pattern image is generated using a TEM SADP simulation program that takes a lattice constant and a unit cell as inputs,
The TEM SADP simulation program is a diffraction pattern image conversion method, characterized in that one of JEMS, QSTEM, abTEM, Ladyne Software Suite, SingleCrystal or Condor. A computer-readable recording medium recording program codes for performing the method of any one of claims 17 to 19.

Images