JP2020204824A - Information processing system and information processing method - Google Patents

Abstract

To automatically determine an appropriate evaluation area size.SOLUTION: An information processing system includes an input unit, an output unit, a storage unit, and a processing unit, calculates a feature amount from a labeling image of an observation image obtained by observing a sample, and evaluates the sample. This system includes: a feature amount extraction unit that calculates the feature amount of each image from multiple observation images or multiple labeling images; and an evaluation area size determination unit that determines an area of the labeling image for evaluating the sample by calculating the feature amount based on a distribution of the feature amount of each image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing technique for extracting and analyzing feature amounts of a material from a microscope image or the like.

In material development, in order to efficiently and experimentally control process conditions, tissue features can be considered as variables between process conditions and material properties. The tissue feature amount can be extracted using a tissue image (for example, a SEM (Scanning Electron Microscope) image). By examining the relationship between this tissue feature and the properties or process conditions of the material, it becomes possible to understand the materials science and support the experimental design for improving the properties.

Patent Document 1 discloses an example of using a machine learning classifier when classifying microscopic images.

Patent Document 2 discloses an SEM including an electron backscatter diffraction (EBSD) detector.

Patent Document 3 discloses the concept of a knowledge discovery device that discovers knowledge about the relationship between an image feature amount and attribute data.

Patent Document 4 discloses "a method of segmenting an image of a biological sample using adaptive classification in order to segment the biological sample into different types of tissue regions" in connection with machine learning. ..

Special Table 2017-520864 Japanese Patent Application Laid-Open No. 2003-121394 Special Table 2004-093006 Gazette Special Table 2017-510792

Conventionally, when extracting a tissue feature amount (hereinafter sometimes referred to as "feature amount") from an image of a material, the user determines by experience which range of the image the feature amount is to be extracted. Was common. The area of the image from which the feature amount is extracted is hereinafter referred to as the "feature amount extraction area".

However, when extracting a feature amount from an image, the result of the feature amount may change depending on the size of the feature amount extraction area (hereinafter, may be referred to as "evaluation area size"). For example, in the case of a material having a homogeneous structure, the feature amount does not change much even if the feature amount extraction region having a relatively narrow evaluation area size is selected from an arbitrary place. However, in the case of a material having a heterogeneous structure, if a feature amount extraction region having a relatively narrow evaluation area size is selected from an arbitrary place, the feature amount may change depending on the place. That is, for example, when evaluated with an area of 100 μm ^2, a homogeneous result can be obtained, but when evaluated with an area of 100 nm ² , the feature amount may change depending on the location. In such a case, the evaluation area size must be increased.

An experienced user can determine the evaluation area size based on the characteristics of the material, but there are individual differences in the judgment, which causes an increase in human cost. In addition, inexperienced users cannot be expected to make an appropriate decision.

Therefore, it is desired that an appropriate evaluation area size can be automatically determined.

One aspect of the present invention is an information processing system that includes an input unit, an output unit, a storage unit, and a processing unit, calculates a feature amount from a labeling image of an observation image in which a sample is observed, and evaluates the sample. .. This system calculates the feature amount of each image from multiple observation images or multiple labeling images, and evaluates the sample by calculating the feature amount based on the feature amount extraction unit and the distribution of the feature amount of each image. It is provided with an evaluation area size determination unit that determines the area of the labeling image to be used.

Another aspect of the present invention is an information processing method executed by an information processing device including an input unit, an output unit, a storage unit, and a processing unit, and calculating a feature amount from a labeling image of an observation image in which a sample is observed. Is. In this method, the feature amount extraction process of calculating the feature amount of each image from a plurality of observed images or a plurality of labeling images of a sample created under the same conditions and the feature amount distribution of each image are used. The evaluation area size determination process, which calculates the feature amount and determines the area of the labeling image for which the sample is evaluated, is performed.

To give a more specific example of the configuration, in the evaluation area size determination process, the first step of preparing a labeling image standardized in advance with a predetermined size and resolution, the first step of combining a plurality of standardized labeling images. Following the second step and the second step, the feature amount is calculated for each of the size evaluation images following the third step and the third step of dividing the plurality of combined labeling images into a plurality of size evaluation images. Following the 4th step and the 4th step, the standard for evaluating the variation situation of the feature amount and returning to the 2nd step based on the variation situation following the 5th step and the 5th step. A sixth step is performed, which determines whether to increase the number of labeled images that have been converted or to determine the area of the labeling image for which the sample is evaluated based on the size of the current size evaluation image.

The appropriate evaluation area size can be determined automatically.

The block diagram of the information processing system which is an Example. The overall flow diagram of the processing by the information processing system of FIG. Image of GUI displayed to identify the sample. The flow chart of the evaluation area size determination process S4. A conceptual diagram showing the concept of combining and splitting images. The graph which shows the deviation of the evaluation area size and the feature amount. A graph showing the deviation between the evaluation area size and the feature amount with the macro size introduced. A conceptual diagram showing the concept of additional combining and splitting of images. An image diagram of the GUI displayed to determine the evaluation area size. Image of GUI displayed to show statistical analysis results. The flow chart of the information retrieval process S9. Image diagram of GUI displayed for process information search processing. A conceptual diagram for evaluating the relationship between the characteristics obtained in the experiment and the characteristics obtained from the model.

The embodiment will be described in detail with reference to the drawings. However, the present invention is not construed as being limited to the description of the embodiments shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or purpose of the present invention.

In the configuration of the invention described below, the same reference numerals may be used in common among different drawings for the same parts or parts having similar functions, and duplicate description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described by adding different subscripts to the same reference numerals. However, if it is not necessary to distinguish between a plurality of elements, the subscript may be omitted for explanation.

Notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number, order, or contents thereof. is not. In addition, numbers for identifying components are used for each context, and numbers used in one context do not always indicate the same composition in other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.
The publications, patents and patent applications cited herein form part of the description herein.
Components represented in the singular form herein shall include the plural form unless explicitly indicated in the context.

<1. Overall system configuration>
FIG. 1 is a block diagram of an information processing system 100 according to an embodiment of the present invention. The information processing system 100 can be configured by, for example, a general server. According to a general server configuration, the information processing system 100 includes an input device, an output device, a storage device, and a processing device.

In the example of FIG. 1, a keyboard or mouse 101 is provided as an input device, an image monitor 102 is provided as an output device, and a CPU (Central Processing Unit) 103 is provided as a processing device. Further, a semiconductor memory 110 and a magnetic disk device 120 are provided as storage devices. Further, it is provided with an input / output interface 104 for inputting / outputting an image signal or the like from the outside. The input / output interface 104 can acquire an image of the material from an observation device such as an SEM directly or via a network. These configurations are connected to each other by a bus and can exchange information, but the bus is not shown.

The above configuration may be configured by a single computer, or any part of the input device, the output device, the processing device, and the storage device may be configured by another computer connected by a network.

In this embodiment, functions such as calculation and control are realized by executing a program stored in the semiconductor memory 110 by the CPU 103 to perform predetermined processing in cooperation with other hardware. The program to be executed, its function, or the means for realizing the function may be referred to as "function", "means", "part", "unit", "module" or the like.

The semiconductor memory 110 stores, as programs, a control unit 111, a segmentation unit 112, an evaluation area size determination unit 113, a feature amount extraction unit 114, a statistical analysis unit 115, a process information retrieval unit 116, and the like. These functions will be described in detail later.

The magnetic disk device 120 has material / process data 121, material property data 122, SEM image data 123, labeling image data 124, labeling model 125, feature amount data 126, statistical analysis result data 127, and process as a database and model. -Characteristic amount-related data 128 and the like are stored.

In the material / process data 121, for example, data on the method of manufacturing the sample and the material used are recorded corresponding to the sample number. The data of the manufactured method includes, for example, processing temperature, processing pressure, processing time, additives, stirring method and the like. Material data includes, for example, the names of the elements used as materials and their ratios.

In the material property data 122, for example, the property data obtained by measuring the sample corresponding to the sample number for specifying the sample of the material created by the experiment is recorded. For the characteristic data, any data such as crystal lattice constant, mass density, hardness, conductivity, thermal conductivity, strength, thermoelectric conversion efficiency, stress, and toughness can be selected.

The SEM image data 123 stores, for example, data of an image obtained by measuring the sample by SEM corresponding to a sample number. If there are multiple images in one sample, each of the multiple images is numbered.

The labeling image data 124 stores data in which the SEM image is labeled in association with the SEM image data 123.

The labeling model 125 stores a trained model used when the labeling image data 124 is automatically created from the SEM image data 123.

The feature amount data 126 stores the feature amount data calculated from the SEM image data 123 or the labeling image data 124 by the feature amount extraction unit 114. The feature amount data is associated with the data of the image from which it is extracted.

The statistical analysis result data 127 stores the data of the result analyzed by the statistical analysis unit 115.

The process / feature amount relationship data 128 stores data indicating the relationship between the process and the feature amount, which is generated from the material / process data 121 and the feature amount data 126.
These contents will be described in detail later.

In this embodiment, functions equivalent to the functions configured by software can be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Such aspects are also included in the scope of the present invention.

<2. Overall processing flow>
FIG. 2 is a diagram showing an overall flow of processing by the information processing system 100 of FIG.
First, the known information of the target material (sample) is input (S1). The information can be input by the user with the keyboard or the mouse 101 each time. Further, the data may be read out from the material / process data 121 and the material property data 122 stored in the magnetic disk apparatus 120 in advance. Alternatively, it may be acquired from the outside via the input / output interface 104.

Next, a plurality of SEM images are acquired (S2). Although it is an SEM image in this embodiment, it may be another type of image acquired by another type of observation device. Other types of images are, for example, optical microscope images, electron microscope images, atomic force microscope images, and the like. The image may be read from the SEM image data 123 stored in the magnetic disk device 120 in advance, or may be input from the observation device connected to the input / output interface 104. Alternatively, it may be acquired via the network connected to the input / output interface 104.

Next, a labeling image corresponding to the acquired SEM image is acquired (S3). The labeling image is an image obtained by analyzing an SEM image and characterizing each part of the image. For example, it is an image that identifies a phase composed of different elements in an SEM image. The labeling image may be read from the labeling image data 124 previously stored in the magnetic disk apparatus 120. Alternatively, the SEM image acquired in (S2) may be subsequently labeled by the segmentation unit 112.

In this embodiment, the labeling processing method is not particularly limited. The user may visually label the SEM image. Alternatively, an automatic classification method using machine learning as disclosed in Patent Document 4 may be used.

Specific examples of the processes S1 to S3 using the GUI (Graphical User Interface) will be described in detail later with reference to FIG.

Next, the evaluation area size determination unit 113 automatically determines the evaluation area size of the image (S4). The method of determining the evaluation area size of the image will be described in detail later with reference to FIGS. 4 to 9.

Next, the feature amount extraction unit 114 extracts the feature amount from the labeling image using the determined evaluation area size (S5).

Next, the statistical analysis unit 115 statistically analyzes the relationship between the extracted features and the material properties (S6).

Next, the statistical analysis unit 115 prioritizes the effect of the feature amount on the material properties (S7).

Next, the statistical analysis unit 115 proposes a key feature amount that needs to be controlled (S8).

Next, the statistical analysis unit 115 determines a keyword and searches for information on process control of key features (S9).

Next, the control unit 111 displays information regarding the process control of the material (S10).

The user examines the displayed information, further manufactures the material if necessary, and adds the data of the sample of the obtained material (S11).

<3. Work example 1 by GUI (specification of material)>
FIG. 3 is an example of a GUI displayed on the image monitor 102 in order to identify the material under the control of the control unit 111. A specific example of the processes S1 to S3 of FIG. 2 will be described with reference to FIG. In this example, the material / process data 121, the material property data 122, the SEM image data 123, the labeling image data 124, the labeling model 125, the feature amount data 126, the statistical analysis result data 127, etc. are stored in the magnetic disk device 120 in advance. Is described as being stored.

In process S1, the "material name" is specified in the pull-down menu 301. Next, the "process No." is specified in the pull-down menu 302 from among the plurality of processes registered in advance for the material. In this example, "material name" is a major classification and "process number" is a minor classification, but other classification methods may be used.

The "process number" specifies the combination of the material and the manufacturing method, and since this information is stored in the database in the material / process data 121, the sample is specified by specifying the "material name" and the "process number". Can be identified. By operating the data acquisition button 303 with the keyboard or mouse 101, data of a desired sample is acquired from the material / process data 121 of the magnetic disk apparatus 120. The operation of GUI buttons is the same as follows.

In the example of FIG. 3, the composition of the material is displayed in the window 304 based on the material / process data 121 of the acquired sample. The material composition is known to the user who created the sample and is stored in the material / process data 121. You can refer to the periodic table if necessary.

The manufacturing conditions are displayed in the window 305. The manufacturing conditions are known to the user who created the sample and are stored in the material / process data 121. For example, temperature, pressure, additives, stirring time and the like. Further, when the material property data 122 has already been acquired in the magnetic disk apparatus 120 for the sample, it is displayed in the window 306. For example, lattice constant, mass density, Vickers strength, conductivity, thermal conductivity and the like.

By operating the observation image acquisition button 307 in the process S2, n SEM images of the sample are acquired from the SEM image data 123 of the magnetic disk apparatus 120. (A part of) the acquired SEM image is displayed in the window 308. It is assumed that the SEM image (and the labeling image) is standardized in advance with a predetermined size and resolution. As an example, a database is created with a standardized size as the minimum unit of an image.

In process S3, a labeling image corresponding to the acquired SEM image is acquired. When the labeling image data 124 already exists in the magnetic disk device 120, the data is called up by operating the labeling image acquisition button 309. (Part of) the recalled labeling image is displayed in window 308.

In FIG. 3, each of the three SEM images and the labeling image are displayed side by side, but the display method and the number of displayed images are arbitrary.

When the magnetic disk device 120 does not have the labeling image data 124, the SEM image is visually labeled on the window 308 by the function of the segmentation unit 112, for example, by the user operating the keyboard or the mouse 101. When the labeling is completed, the labeling information is confirmed or determined by operating the button 310. The labeling information is, for example, the name and color of each phase (region). For example, "Phase A, RGB (255,200,100)" is determined. The labeling image for which labeling has been determined is displayed in the window 308 and recorded as the labeling image data 124 on the magnetic disk device 120.

Further, as described above, when the machine learning technique is applied to automatically perform the labeling, the segmentation unit 112 calls the labeling model 125 by operating the labeling execution button 311 and automatically performs the labeling. The labeling model 125 is, for example, a DNN (Deep Neural Network) learned by supervised learning using a pair of an SEM image and a labeling image labeled by the user as a teacher image.

By operating the image data button 312, data such as observation conditions of each SEM image can be recalled, and can be modified or confirmed. Data such as observation conditions include, for example, an image name, an observation device ID, a worker, an accelerating voltage, a WD (working distance), a magnification, a scan time for image storage, and the like, and these are metadata of SEM image data 123.

After performing the above processing and the user can grasp the contents of the sample, the evaluation area size determination processing S4 proceeds by operating the button 313.

<4. Evaluation area size determination process>
FIG. 4 is a detailed flow of the evaluation area size determination process S4 executed by the evaluation area size determination unit 113. In the processes S1 to S3, the sample SEM image data and the labeling image data created by the material / process specified by the user are read into the working area of the semiconductor memory 110.

In process S4-1, the evaluation area size determination unit 113 determines the method of combining the image data. The combination method refers to how many images are combined (or added). For example, three images may be combined horizontally, or nine images may be combined in a 3x3 manner. The combination method may be determined by default, may be specified by the user each time, or may be selected by the user from the options prepared in advance. In the following example, an example of combining three images horizontally will be described.

The images may be combined and divided using an SEM image or a labeling image. Since both are compatible, they can be replaced. In the following, a labeling image will be used for explanation.

In the process S4-2, a plurality of images are combined by a determined method. Combining is the joining of multiple images.

In the process S4-3, the combined image is evenly divided into three size evaluation images (hereinafter, may be referred to as "evaluation images"). The reason for dividing into three is to calculate these deviations later.

In the process S4-4, the feature amount extraction unit 114 is called to calculate the feature amount of the labeling image corresponding to the three evaluation images. When the processing up to S4-3 is performed on the labeling image, the feature amount may be calculated from the labeling image. When the processing up to S4-3 is performed on the SEM image, the corresponding labeling image is read from the labeling image data 124.

The feature quantities of the labeling image are, for example, the number of phases, the area ratio of each phase, the shape information of each phase, the area of each phase (mean value, maximum value, minimum value, variation), and the maximum linear length of each phase (average value). Value, maximum value, minimum value, variation), etc. may be selected arbitrarily. In this embodiment, the area ratio of each phase, which is considered to have a large influence on the material properties, will be described as an example.

After the feature amount is calculated, the evaluation area size determination unit 113 calculates the deviation of the feature amount. Here, the deviation is
Deviation =
(std (Total area of the feature)) / (ave (Total area of the feature))
Defined in. std means standard deviation, ave means mean, and "Total area of the feature" means the entire area of interest.

In process S4-5, the calculated deviation is compared with a predetermined condition. In a typical process, the deviation is compared with a predetermined threshold to determine if it is less than the threshold. If it is equal to or more than the threshold value, the process proceeds to process S4-6, and if it is smaller than the threshold value, the process proceeds to S4-7.

In the process S4-6, an image is further added to the images combined in the process S4-2, and a plurality of images are combined. After that, the processes S4-3 to S4-5 are repeated.

In the process S4-7, when the deviation is smaller than the threshold value, the size of the evaluation image divided in the process S4-3 is proposed to the user as the evaluation area size by an image monitor or the like.

In the process S4-8, the next process is advanced according to the evaluation area size determined by the user.

FIG. 5 shows the concept of combining and dividing the labeling image in the processing S4-2 and the processing S4-3. Three labeling images 501 whose size is standardized as shown in (A) are combined horizontally as shown in (B). After that, as shown in (C), the evaluation image 502 is generated by dividing it into three equal parts. In FIG. 5, three labeling images 501 are combined, but two or four or more may be used. When calculating the deviation, three evaluation images 502 are required, but when using another evaluation method described later, the number may not be three or more. In the example of FIG. 5, the evaluation area size pX ² of the evaluation image 502 is W ₁₁ × W ₁₂ .

The labeling images 501-1, 501-2, 501-3 may correspond to arbitrary regions on the sample, and need not correspond to continuous regions. For example, it may be randomly selected from SEM image data of the same sample or the same process stored in the magnetic disk apparatus 120. In the labeling image 501, labeling is performed, for example, phases A, B, and C.

A method of determining the evaluation area size according to the present embodiment will be described with reference to FIG. Here, the area ratios of a plurality of phases A, B, and C are used as the feature amount of the labeling image. The phases A, B, and C may be distinguished by their contained elements or by other properties such as crystal structure.

Figure 6 is evaluated in abscissa area size pX ^2, shows the deviation of the characteristic quantity on the vertical axis. The threshold used in S4-5 shown in _{dev th.} It is assumed that the threshold value dev _th is predetermined by the user and stored by the evaluation area size determination unit 113.

As shown in FIG. 6, in general, as the evaluation area size increases, the feature amount is averaged, so that the variation (deviation in this example) of the feature amount between the evaluation images becomes smaller. In this embodiment, all of the evaluation area size A _det deviation of the phase is smaller than the threshold value dev _th, so that the proposed user. That is, it is possible to propose the minimum evaluation area size that can appropriately represent the characteristics of the sample. The threshold value may be set for each phase.

FIG. 7 describes a method for determining the evaluation area size when the macro size is introduced. In Figure 7, the evaluation area size on the horizontal axis pX ^2, shows the deviation of the characteristic quantity on the vertical axis. In this example, increasing the evaluation area size, it is assumed that the deviation does not fall below the threshold dev _th. In this case, along with the threshold _{dev th} deviation, it defines the macro size A _MC. While the deviation is not below the threshold dev _th, the evaluation area size A _det is when it reaches the macro size A _MC, without increasing the more evaluation area size, the evaluation area size macro size.

FIG. 8 shows the concept of additional processing of SEM image data in processing S4-6 when the calculated deviation is equal to or greater than a predetermined threshold value. As compared with FIG. 5, six additional labeling images 501add are added to the labeling images 501-1, 501-2, and 501-3 by the process S4-6 as shown in (A). Combine this as shown in (B). After that, as shown in (C), the evaluation image 502add is generated by dividing it into three equal parts. It should be noted that how to add / combine images is defined in process S4-1.

In the evaluation image 502 of FIG. 5, the evaluation image 502-2 in the middle of the three does not contain the characteristic phases B and C, and is not suitable for evaluating the feature amount of the material. In the evaluation image 502add in FIG. 8, each evaluation image contains the phases A, B, and C substantially evenly, and the feature amount of the material can be appropriately evaluated from the evaluation image. In particular, as shown in FIGS. 5 and 8, when the presence of phases B and C having a small area affects the characteristics of the material with respect to the phase A occupying most of the area (for example, 90% or more). It is important that these phases are included in the feature extraction area. In this embodiment, an appropriate size of the feature amount extraction region can be automatically determined.

<5. Supplementary explanation>
(1) In the above description, in the process S4-2, a plurality of images are combined, but if the size of the standardized image is large, the process S4-2 may be omitted.
(2) In the above description, in the process S4-3, the combined image is evenly divided into three evaluation images, but in the processes S4-4 to S4-5, the following other images are used instead of the deviation. When the evaluation index is used, it may be 2 or more. For example
A) The variance δ2 of the feature amount of the evaluation image becomes smaller than the threshold value.
B) The difference between the feature amount of each evaluation image and the average value of the feature amount is all smaller than the threshold value.
C) The standard deviation δ of the feature quantity becomes smaller than the threshold value.
d) The differences between the features of the evaluation image are all smaller than the threshold value.
(3) In the above description, the labeling images are combined and divided in the processes S4-2 to S4-3, but the corresponding SEM images may be combined and divided.
(4) In the above description, the feature amount of the labeling image is calculated in the process S4-4, but the feature amount of the SEM image may be used instead. The feature amounts of the SEM image include image frequency, average brightness, average contrast, and the like. However, it is preferable to use the labeling image because the labeling image directly represents the features of the sample to be analyzed.

<6. Work example 2 by GUI (determination of evaluation area size)>
FIG. 9 is an example of a GUI displayed on the image monitor 102 under the control of the control unit 111. By operating the button 313 of FIG. 3, the above-mentioned processes S4-1 to S4-6 are performed, and then the screen of FIG. 9 is displayed to determine the evaluation area size.

A specific example of the processes S4-7 to S4-8 of FIG. 4 will be described with reference to FIG. The recommended evaluation area size 901 obtained by the processes S4-1 to S4-6 of FIG. 4 includes information such as the process number 902 for specifying the sample and the sample code 903 created in the process by the function of the control unit 111. , Displayed in window 904. In this example, since the recommended evaluation area size 901 and the like are displayed for a plurality of samples, the processes of FIGS. 2 and 4 are also performed for the plurality of samples.

Although the size of image data is handled in pixel units in the system, it is preferable to convert it into actual size data and display it as appropriate. Here, the recommended evaluation area size 901 is calculated from pX ² = W ₁₁ × W ₁₂ (see FIGS. 5 and 8). The default value of the evaluation area size 905 may be the same as the recommended evaluation area size 901. The user changes the evaluation area size 905 as needed.

If the evaluation area size 905 is not optimized, a warning is displayed in the window 906. In this example, the evaluation area size of process number 1 is a size whose deviation does not satisfy the threshold value in the determination of the process S4-5 of FIG. In this case, the user can choose to ignore with button 907 and continue, or with button 908 to add more images to increase the size. When the size is increased, the process returns to S4-6 of FIG.

When the deviation does not satisfy the threshold value in the determination of the process S4-5 of FIG. 4, the recommended evaluation area size 901 is proposed in the process S4-7 by defining the macro size A _MC as described above. If you are. User by buttons 908, can increase the size ignoring macro size A _MC. Alternatively, the user can enter an evaluation area size 905 that is smaller than the recommended evaluation area size 901. Again, a warning is displayed in window 906, but the user can ignore it with button 907 and continue.

The window 904 also shows the feature amount 909 calculated from the labeling image of each sample. As these feature quantities, the feature quantities calculated in the process S4-4 of FIG. 4 may be used. In this example, the area ratios of phases A, B, and C, the average size of each phase, and the like are shown as the feature amount 909. In this system, the area ratio of the phase is represented by "% (name of the phase)". When the evaluation area size 905 is changed, the user can recalculate the feature amount using the reanalysis button 910.

When the evaluation area size 905 is finally determined, the user presses the feature amount extraction button 911 to start the actual feature amount extraction process (S5 in FIG. 2). In the actual feature amount extraction process S5, it is desirable to extract the feature amount in more detail and in a multifaceted manner than the feature amount calculated in the process S4-4 of FIG. The extracted feature amount can be displayed in the window 904. In this embodiment, it is assumed that the feature amount extraction unit 114 stores algorithms for extracting various feature amounts, and the control unit 111 calls them. The extracted feature amount is stored in the magnetic disk device 120 as the feature amount data 126.

With the check button 912, an evaluation area size based on typical conditions can be applied. In the example of FIG. 9, samples having a plurality of process numbers are displayed, and the evaluation area size is calculated for each sample. However, if it is known in advance that the process conditions of the samples are similar, the user can use the check button 912 to apply the evaluation area size of one sample to the other samples. For example, when there is little organizational variation within the range of process conditions, select a representative condition (for example, Process No. 1) from the process conditions, determine the evaluation size of the image, and then use the same image evaluation size for all process conditions. Do.

When the feature amount extraction process S5 is completed, the feature amount effectiveness extraction process (S6 to S10 in FIG. 2) can be started by pressing the feature amount effectiveness evaluation button 913. Effectiveness of feature amount In the extraction process, the correlation analysis between the feature amount and the material property is performed. The correlation analysis is performed by the correlation calculation unit provided as a part of the algorithm of the feature amount extraction unit 114. As a result of the correlation analysis, it is possible to know which feature amount has a large effect on the material properties. That is, it is possible to specify an important feature amount for controlling the characteristic. Next, we will search for what kind of processes and materials are available to control the features. By examining the search results by the user, the next experimental plan can be formulated and a new sample can be obtained.

<7. Analysis of the relationship between features and material properties>
As a result of processing S5, feature data 126 could be prepared for each sample. The feature amount data 126 can be cross-referenced with the material / process data 121, the material property data 122, the SEM image data 123, and the labeling image data 124. For example, the process number and the sample code are used as keys to correspond to a specific sample. You can identify the data to be used. Therefore, for example, it is possible to draw a graph in which the feature amount is displayed on the horizontal axis and the material property is displayed on the vertical axis.

With reference to FIG. 2 again, the statistical analysis unit 115 analyzes the relationship between the feature amount and the material property in the process S6 based on the above data. As the analysis method, known principal component analysis or linear analysis can be used and is not particularly limited. Further, the statistical analysis unit 115 prioritizes the effect (correlation degree) of the feature amount on the material properties in the process S7. For example, the priority is increased in the order of the correlation coefficient. Then, in the process S8, the statistical analysis unit 115 proposes a feature amount having a high priority as an important feature amount that needs to be controlled.

As the result of the statistical analysis, the correlation (for example, the correlation coefficient) of the feature amount with respect to each material property is recorded in the magnetic disk apparatus 120 as the statistical analysis result data 127.

FIG. 10 is an example of a GUI showing the analysis results of the processes S6 to S8 displayed on the image monitor 102 under the control of the control unit 111. The user can select the material properties to be analyzed by the pull-down menu 1001. Here, Vickers strength is selected. As a result, on the left side of the window 1002, graphs showing the correlation between the Vickers intensity and the feature amount are displayed in descending order of the degree of correlation. On the right side of the window 1002, as more detailed information, a graph showing the relationship between the Vickers intensity and the feature amount is displayed for the top three feature amounts having a high degree of correlation. For example, these three features will be proposed as candidates for key features that need to be controlled. The user can change the number of features to be displayed by the pull-down menu 1003.

From the above, the user can know what the feature amount is presumed to be important for controlling the Vickers strength. Results such as the proposed features are stored in the magnetic disk device 120 as statistical analysis result data 127 by pressing the record button 1004. Next, the user uses the process information search button 1005 to start the process information search process S9 and the result display process S10.

The sample addition button 1106 is for recalculating the relationship between the feature amount and the material property when new sample data is input.

<8. Process information search process>
FIG. 11 is a flow chart showing details of the process information search processes S9 and S10. These processes are executed by the process information retrieval unit 116.
FIG. 12 is an example of a GUI displayed on the image monitor 102 under the control of the control unit 111 in the process information search process S9. This will be described with reference to FIGS. 11 and 12.

In FIG. 12, the user inputs the material properties to be improved and the purpose or the content to be controlled (“increase”, “decrease”, etc.) in the window 1201. The content input by the purpose addition button 1202 is determined as a target (S9-1).
The process information retrieval unit 116 determines a keyword based on the word input in the process S9-1, for example, by a known syntactic analysis. In the window 1203 of FIG. 12, "Vickers" and "strength" are displayed as keywords as examples (S9-2). These keywords are defined in the system of the example, and the variable names can be used as they are.

The process information retrieval unit 116 correlates important feature quantities and Vickers intensity from the statistical analysis result data 127 of the magnetic disk device 120 based on the keywords “Vickers” and “intensity” to control the “Vickers intensity”. Call the data of. As a specific example, for example, in FIG. 10, data showing the relationship between the top nine feature quantities extracted as the feature quantities highly related to the Vickers intensity and the "Vickers intensity" is called. Then, using the features, a characteristic prediction model is generated, and the model with the best accuracy is selected.

From the statistical analysis result data 127, a feature amount having a high correlation with a predetermined characteristic can be identified, but it is unknown what kind of effect the feature amount has on the characteristic. Therefore, a characteristic prediction model showing the relationship between the feature quantity and the characteristic is created. The characteristic prediction model is, for example, a linear regression model, and a model that clearly shows the relationship between the feature quantity and the characteristic is created by using a known method with the feature quantity as an explanatory variable and the characteristic as an objective variable.

For this purpose, for example, model candidates are created by using highly correlated features one by one as explanatory variables. When the feature quantity is 9 in FIG. 10, 9 model candidates are created. Alternatively, model candidates are created using the features as multiple explanatory variables. After creating a plurality of model candidates, select the model that most accurately expresses the characteristics (S9-3).

FIG. 13 shows a concept for evaluating the relationship between the property obtained by the experiment (Experimental Property) and the property obtained from the model (Predicted Property) for the three models. When the highly relevant feature quantities are X1, X2, X3, it is assumed that the model 1 uses only X1, the model 2 uses X1, X2, and the model 3 uses X1, X2, X3 to represent the characteristic Y. .. In this example, the model with model number 3 has the smallest error and well represents the experimental characteristics, so model number 3 is selected as the characteristic prediction model.

Next, the process information retrieval unit 116 determines the feature amount used in the model selected in the process S9-3 as a key feature amount. In the case of FIG. 13, X1, X2, and X3 are determined as key feature quantities (S9-4).

Next, the process information retrieval unit 116 determines a keyword based on the feature amount determined in the process S9-4 (S9-5). For example, the feature quantities X1, X2, and X3 are "% phase 1", "Average size of phase 1", and "% phase 2" shown in FIG. These data representations are defined in the system of examples, where "% phase X" indicates the area ratio of the phase consisting of element X and "Average size of phase X" indicates the area ratio of the phase consisting of element X. Indicates the average size (eg area). The process information retrieval unit 116 generates a keyword from these features. For example, "% phase 1", "% phase 2", and "average size of phase 1". In addition, it is assumed that 1 and 2 of phase 1 and phase 2 indicate some element name (element 1, element 2). The material name "ABC" of the sample was previously extracted as a keyword (S9-2).

After that, the relationship between the process and the feature amount is searched, but the process branches depending on whether or not the system has the relationship data between the process and the feature amount (S9-6). When searching from the relationship between the process condition and the feature amount, the search is performed using the data of the magnetic disk device 120, and the relationship between the process condition and the feature amount is displayed. In addition, when searching from external information, related information (for example, literature) is displayed.

<8-1. If the system has process information>
In the material / process data 121 and material property data 122 stored inside the magnetic disk device 120, when the feature amount, material / process, and property of the sample are associated, a key for obtaining a predetermined property is obtained. The expected change in the feature amount (for example, the feature amount X1, X2, X3 in FIG. 13) is determined (S9-7A). For this purpose, in the characteristic prediction model selected in (S9-3), the change required for the feature amount is determined in order to obtain the characteristic change specified in the window 1201 of FIG. For example, in the model of No. 3 in FIG. 13, in order to increase the characteristic Y, the feature quantities X1 and X3 are increased and X2 is decreased.

In the process S9-8A, the keyword is determined based on the change in the feature amount determined in S9-7A. For example, "feature amount X1 * increase", "feature amount X2 * decrease", and "feature amount X3 * increase". The process information search unit 116 searches for the material / process data 121 and the feature amount data 126 by operating the search button 1204 using the keywords defined in the processes S9-2, S9-5, and S9-8A.

For example, if the relationship between the sample of a specific material and the process can be specified in the material / process data 121, and the relationship between the sample of the specific material and the feature amount can be specified in the feature amount data 126, the process and the feature amount can be specified. Data showing changes can be generated. Such data may be generated from the material / process data 121 and the feature amount data 126 by batch processing and stored in the magnetic disk apparatus 120 as the process / feature amount relationship data 128. At this time, for example, it is desirable to name the data as "relationship data between the feature amount X1 and the process P1" and add tags such as "increase" and "decrease".

Such process / feature amount relation data 128 is searched by the above-mentioned keyword.

In window 1205 of FIG. 12, a graph showing the relationship between the feature amount extracted as a result of the search and the process is displayed together with arrows indicating "increase" and "decrease" of the feature amount. In this example, the horizontal axis shows the number of times the material is agitated, and the vertical axis shows various features. This allows the user to know which process should be modified to bring about a given change in properties in the material.

<8-2. If the system does not have process information>
If the system does not have the data related to the process and the feature amount, the process information is searched using the existing database.

In the absence of data showing the relationship between material properties and processes, a general representation of the keyword is determined (S9-7B). As a generalization method, for example, a thesaurus is used to change the keyword "ABC" defined in the process S9-2 to "alloy" or "polymer". Alternatively, change from "Vickers strength" to the broader keyword "strength" or "hardness". Further, the keyword defined in the process S9-5, for example, "% phase 1" is changed to "element 1" and "area ratio". Since the keywords used in the window 1203 may use expressions peculiar to this system, the corresponding general keywords are shown in the window 1206. Keyword conversion can be performed using a correspondence table or a dictionary. For example, since "% phase1" is not a general notation, it is converted into "element 1" and "area ratio".

Further, in the process S9-7B, the keyword may be edited or added by the user on the screen shown in FIG. The final keyword may be decided after the user confirms it. Alternatively, the user confirmation may be omitted.

Then, it connects to an existing general search engine or research database via the input / output interface 104 (S9-8B), searches for information on process control of microstructure features, and displays it in window 1207 (S9-10B). ). When the sample data increases, the sample addition button 1208 is used to recalculate.

According to the examples described above, in the material development, the evaluation area size of each sample is determined by using the structure measuring device or the structure image, and the tissue feature amount is extracted within the evaluation size, and the most material property is obtained. It is possible to provide a material structure analysis system that analyzes a tissue feature with a high degree of influence and provides information on the control of the tissue feature. By determining the evaluation area size of the tissue image, a highly reliable feature amount can be extracted even when the amount of heterogeneous phase is small. In addition, it can provide information on the control of important tissues for a given characteristic. Further, when the image data is input and the model for obtaining a predetermined output is machine-learned, the area of the determined evaluation area size is used as the area of the input image data, which is appropriate as the output of the model. Value can be obtained.

Control unit 111, segmentation unit 112, evaluation area size determination unit 113, feature quantity extraction unit 114, statistical analysis unit 115, process information retrieval unit 116, material / process data 121, material characteristic data 122, SEM image data 123, labeling Image data 124, labeling model 125, feature amount data 126, statistical analysis result data 127, process / feature amount relationship data 128.

Claims (17)

An information processing system that includes an input unit, an output unit, a storage unit, and a processing unit, calculates a feature amount from a labeling image of an observation image in which a sample is observed, and evaluates the sample.
A feature amount extraction unit that calculates the feature amount of each image from the plurality of the observed images or the plurality of the labeling images, and
An evaluation area size determination unit is provided, which calculates the feature amount based on the distribution of the feature amount of each image and determines the area of the labeling image for evaluating the sample.
Information processing system. The evaluation area size determination unit
The area of the labeling image is determined based on at least one of the deviation of the feature amount, the variance of the feature amount, the difference between the average value of each feature amount and the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts. ,
The information processing system according to claim 1. The evaluation area size determination unit
The labeling image until at least one of the deviation of the feature amount, the variance of the feature amount, the difference between the average value of each feature amount and the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts becomes smaller than a predetermined threshold value. Increase the area,
The information processing system according to claim 1 or 2. The labeling image is an image in which each portion in the image of the observation image is divided into a plurality of phases based on at least one of the contained element and the crystal structure.
The information processing system according to any one of claims 1 to 3. The feature amount extraction unit calculates at least one of the area ratio, average area, and shape information of each phase of the labeling image as a feature amount.
The information processing system according to claim 4. The feature amount extraction unit calculates the feature amount of each image from the plurality of the observed images or the plurality of the labeling images obtained from the samples of the same process.
The information processing system according to any one of claims 1 to 5. The evaluation area size determination unit
When evaluating the labeling image of the observation image obtained by observing the samples by different manufacturing processes and the difference in the manufacturing process between the samples is within a predetermined range,
The area of the labeling image that evaluates the sample from one manufacturing process is also applied to the labeling image that evaluates the sample from another manufacturing process.
The information processing system according to any one of claims 1 to 6. Has a correlation calculation unit
The correlation calculation unit
The feature amount of the labeling image of the observation image obtained by observing the sample by a different manufacturing process is input.
Using the material properties of the samples from the different manufacturing processes as inputs
The degree of influence of the feature amount on the material property is calculated.
The information processing system according to any one of claims 1 to 7. It has a process information search unit
The process information retrieval unit
A key feature amount selected based on the degree of influence on the material properties is input, and a keyword for search is determined based on the key feature amount.
The information processing system according to claim 8. The storage unit
The process / feature data that records the relationship between the process conditions of the sample by different manufacturing processes and the feature amount of the labeling image of the observation image in which the sample is observed is stored.
Search for the process / feature data with the keyword,
The information processing system according to claim 9. It is an information processing method that is executed by an information processing device including an input unit, an output unit, a storage unit, and a processing unit, and calculates a feature amount from a labeling image of an observation image in which a sample is observed.
A feature amount extraction process for calculating a feature amount of each image from a plurality of the observed images or a plurality of the labeled images of a sample created under the same conditions.
An evaluation area size determination process that calculates the feature amount based on the distribution of the feature amount of each image and determines the area of the labeling image for evaluating the sample.
Information processing method to perform. In the evaluation area size determination process
The first step of preparing a labeling image standardized in advance with a predetermined size and resolution,
A second step of combining a plurality of the standardized labeling images,
Following the second step, a third step of dividing a plurality of combined labeling images into a plurality of size evaluation images,
Following the third step, the fourth step of calculating the feature amount for each of the size evaluation images,
Following the fourth step, the fifth step of evaluating the state of variation in the feature amount,
Following the fifth step, based on the variability situation, the number of standardized labeling images to be combined by returning to the second step is increased, or depending on the size of the current size evaluation image. A sixth step, determining whether to determine the area of the labeling image to evaluate the sample.
11. The information processing method according to claim 11. In the fifth step,
Calculate the deviation of the feature, the variance of the feature, the difference between each feature and the mean value of the feature, the standard deviation of the feature, and the difference between the features.
The information processing method according to claim 12. In the sixth step,
When the deviation of the feature amount, the variance of the feature amount, the difference between the average value of each feature amount and the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts become smaller than a predetermined threshold, the current size evaluation image Determines the area of the labeling image on which the sample is evaluated by the size of
The information processing method according to claim 12 or 13. The seventh step of calculating the feature amount from the labeling image of the observation image obtained by observing a plurality of samples taken under different manufacturing process or composition conditions by applying the area of the labeling image determined in the sixth step. ,
Eighth step of preparing data on material properties of a plurality of samples imaged under different manufacturing processes or compositions.
Ninth step of performing a correlation analysis between the feature amount and the material property and evaluating the degree of influence of the feature amount on the material property.
A tenth step of determining a keyword for data retrieval based on a feature amount applicable to a target value of the material property based on the evaluation result of the influence degree of the feature amount.
The information processing method according to any one of claims 12 to 14. When machine learning a model to obtain a predetermined output by inputting image data
As the area of the input image data, the area of the labeling image for evaluating the determined sample is used.
The information processing method according to any one of claims 11 to 15. The feature amount extraction unit
By applying the area of the labeling image determined by the evaluation area size determination unit, the feature amount is calculated from the labeling image of the observation image obtained by observing a plurality of samples.
The information processing system according to any one of claims 1 to 10.

Images