JP7247774B2 - Information processing system and information processing method - Google Patents

Description

TECHNICAL FIELD The present invention relates to an information processing technology for extracting and analyzing material feature amounts from a microscope image or the like.

In materials development, in order to efficiently control process conditions experimentally, we can consider microstructure features as variables between process conditions and material properties. A tissue feature amount can be extracted using a tissue image (for example, a SEM (Scanning Electron Microscope) image). By examining the relationship between this structural feature quantity and material properties or process conditions, we can understand material science and support experimental plans for property improvement.

Patent Document 1 discloses an example of using a machine learning classifier when classifying microscope images.

US Pat. No. 6,200,000 discloses an SEM with an electron backscatter diffraction (EBSD) detector.

Patent Document 3 discloses the concept of a knowledge discovery device that discovers knowledge about the relationship between image feature amounts and attribute data.

In the context of machine learning, US Pat. .

Special Table 2017-520864 Japanese Patent Application Laid-Open No. 2003-121394 Special table 2004-093006 Special Table 2017-510792

Conventionally, when extracting a tissue feature quantity (hereinafter sometimes referred to as a "feature quantity") from an image of a material, it is up to the user to determine from which range of the image the feature quantity is to be extracted based on experience. was common. The area of the image from which the feature amount is extracted is hereinafter referred to as a "feature amount extraction area".

However, when extracting a feature amount from an image, the result of the feature amount may change depending on the size of the feature amount extraction region (hereinafter sometimes referred to as "evaluation area size"). For example, if the material has a homogeneous structure, even if a feature amount extraction region with a relatively narrow evaluation area size is selected from an arbitrary location, the feature amount does not change much. However, if the material has an inhomogeneous structure and a feature amount extraction region with a relatively narrow evaluation area size is selected from an arbitrary location, the feature amount may vary depending on the location. For example, evaluation with an area of 100 μm ² yields uniform results, but evaluation with an area of 100 nm ² may result in different feature amounts depending on the location. In such a case, the evaluation area size must be increased.

An experienced user can determine the size of the evaluation area based on the characteristics of the material, but there are individual differences in judgment, which increases the personnel cost. Also, inexperienced users cannot be expected to make good decisions.

Therefore, it is desirable to be able to automatically determine an appropriate evaluation area size.

One aspect of the present invention is an information processing system that includes an input unit, an output unit, a storage unit, and a processing unit, calculates a feature amount from a labeling image of an observed image obtained by observing a sample, and evaluates the sample. . This system consists of a feature extracting unit that calculates the feature of each image from multiple observed images or multiple labeling images, and a feature extractor that calculates the feature based on the distribution of the feature of each image and evaluates the sample. and an evaluation area size determination unit that determines the area of the labeling image to be processed.

Another aspect of the present invention is an information processing method that is executed by an information processing apparatus that includes an input unit, an output unit, a storage unit, and a processing unit, and calculates a feature amount from a labeling image of an observed image obtained by observing a sample. is. In this method, based on the feature amount extraction process of calculating the feature amount of each image from a plurality of observation images or a plurality of labeling images regarding samples created under the same conditions, and the distribution of the feature amount of each image, and evaluation area size determination processing for determining the area of the labeling image for evaluating the sample by calculating the feature amount.

As a more specific example of the configuration, in the evaluation area size determination process, a first step of preparing a labeling image standardized in advance with a predetermined size and resolution, a second step of combining a plurality of standardized labeling images, Step 2, following the second step, a third step of dividing a plurality of combined labeling images into a plurality of size evaluation images, following the third step, calculating a feature amount for each of the size evaluation images. 4th step, following the 4th step, a 5th step of evaluating the variation situation of the feature quantity, following the 5th step, based on the variation situation, return to the 2nd step and combine standards A sixth step is to determine whether to increase the number of labeled images that have been converted, or to determine the area of the labeling images for evaluating the sample by the size of the current size evaluation image.

An appropriate evaluation area size can be determined automatically.

1 is a block diagram of an information processing system that is an embodiment; FIG. FIG. 2 is an overall flowchart of processing by the information processing system of FIG. 1; The image diagram of GUI displayed for specifying a sample. FIG. 10 is a flowchart of evaluation area size determination processing S4; 4 is a conceptual diagram showing the concept of combining and dividing images; FIG. FIG. 10 is a graph showing deviations between evaluation area sizes and feature amounts; FIG. 10 is a graph showing the deviation between the evaluation area size and the feature amount with the introduction of the macro size; FIG. 4 is a conceptual diagram showing the concept of additional combination and division of images; FIG. 4 is an image diagram of a GUI displayed for determining the evaluation area size; The image figure of GUI displayed in order to show a statistical analysis result. The flowchart of information search processing S9. FIG. 4 is an image diagram of a GUI displayed for process information search processing; Conceptual diagram for evaluating the relationship between characteristics obtained from experiments and characteristics obtained from models.

Embodiments will be described in detail with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will easily understand that the specific configuration can be changed without departing from the idea or gist of the present invention.

In the configuration of the invention described below, the same reference numerals may be used in common for the same parts or parts having similar functions between different drawings, and redundant description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described with the same reference numerals and different suffixes. However, if there is no need to distinguish between multiple elements, the subscripts may be omitted.

Notations such as “first”, “second”, “third” in this specification etc. are attached to identify the constituent elements, and do not necessarily limit the number, order, or content thereof isn't it. Also, numbers for identifying components are used for each context, and numbers used in one context do not necessarily indicate the same configuration in other contexts. Also, it does not preclude a component identified by a certain number from having the function of a component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings and the like.
All publications, patents and patent applications cited herein are hereby incorporated by reference into this description.
Elements presented herein in the singular shall include the plural unless the context clearly dictates otherwise.

<1. Overall system configuration>
FIG. 1 is a block diagram of an information processing system 100 that is an embodiment of the present invention. The information processing system 100 can be configured by, for example, a general server. Following a general server configuration, the information processing system 100 includes input devices, output devices, storage devices, and processing devices.

The example in FIG. 1 includes a keyboard and mouse 101 as input devices, an image monitor 102 as an output device, and a CPU (Central Processing Unit) 103 as a processing device. In addition, a semiconductor memory 110 and a magnetic disk device 120 are provided as storage devices. Further, an input/output interface 104 is provided for inputting/outputting image signals from the outside. The input/output interface 104 can acquire an image of the material from an observation device such as an SEM directly or via a network. These components are connected to each other by a bus and can exchange information, but the bus is not shown.

The above configuration may be composed of a single computer, or arbitrary portions of the input device, output device, processing device, and storage device may be composed of other computers connected via a network.

In the present embodiment, functions such as calculation and control are realized by executing a program stored in the semiconductor memory 110 by the CPU 103, thereby performing predetermined processing in cooperation with other hardware. A program to be executed, its function, or a means for realizing that function may be called a "function", a "means", a "unit", a "unit", a "module", or the like.

The semiconductor memory 110 stores, as programs, a control unit 111, a segmentation unit 112, an evaluation area size determination unit 113, a feature amount extraction unit 114, a statistical analysis unit 115, a process information search unit 116, and the like. These functions will be described in detail later.

The magnetic disk device 120 stores material/process data 121, material property data 122, SEM image data 123, labeling image data 124, labeling model 125, feature amount data 126, statistical analysis result data 127, process - The feature quantity relationship data 128 and the like are stored.

The material/process data 121 records, for example, data on the method of manufacturing the sample and the material used in correspondence with the sample number. The data on the manufacturing method include, for example, processing temperature, processing pressure, processing time, additives, stirring method, and the like. The material data includes, for example, the names of the elements that make up the materials and their proportions.

In the material property data 122, for example, property data obtained by measuring the sample corresponding to the sample number specifying the material sample created by experiment is recorded. Any data such as crystal lattice constant, mass density, hardness, electrical conductivity, thermal conductivity, strength, thermoelectric conversion efficiency, stress, and toughness can be selected as the characteristic data.

The SEM image data 123 stores image data obtained by measuring the sample with the SEM, for example, corresponding to the sample number. If there are multiple images for one sample, each of the multiple images is numbered.

The labeling image data 124 stores data obtained by labeling the SEM image in association with the SEM image data 123 .

The labeling model 125 stores a trained model used when automatically creating the labeling image data 124 from the SEM image data 123 .

The feature amount data 126 stores feature amount data calculated from the SEM image data 123 or the labeling image data 124 by the feature amount extraction unit 114 . The feature amount data is associated with the image data from which it was extracted.

Statistical analysis result data 127 stores data of the results of analysis by statistical analysis unit 115 .

The process/feature quantity relationship data 128 stores data indicating the relationship between the process and the feature quantity generated from the material/process data 121 and the feature quantity data 126 .
These contents will be described later in detail.

In this embodiment, functions equivalent to those configured by software can also be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). Such aspects are also included in the scope of the present invention.

<2. Overall processing flow>
FIG. 2 is a diagram showing the overall flow of processing by the information processing system 100 of FIG.
First, known information about the target material (sample) is input (S1). Information can be input by the user using the keyboard or mouse 101 each time. Alternatively, data may be read from the material/process data 121 and material property data 122 stored in the magnetic disk device 120 in advance. Alternatively, it may be obtained from the outside via the input/output interface 104 .

Next, a plurality of SEM images are acquired (S2). Although the SEM image is used in this embodiment, other types of images acquired by other types of observation devices may be used. Other types of images are, for example, optical microscope images, electron microscope images, atomic force microscope images, and the like. The image may be read from the SEM image data 123 stored in the magnetic disk device 120 in advance, or may be input from an observation device connected to the input/output interface 104 . Alternatively, it may be acquired via a network connected to the input/output interface 104 .

Next, a labeling image corresponding to the acquired SEM image is acquired (S3). A labeling image is an image obtained by analyzing an SEM image and characterizing portions of the image. For example, an image identifying phases composed of different elements in an SEM image. The labeling image may be read from the labeling image data 124 stored in the magnetic disk device 120 in advance. Alternatively, the SEM image acquired in (S2) may be subjected to labeling processing by the segmentation unit 112 subsequently.

In this embodiment, the method of labeling processing is not particularly limited. A user may label the SEM image visually. Alternatively, an automatic classification method using machine learning, such as that disclosed in Patent Document 4, may be used.

A specific example of the processes S1 to S3 using a GUI (Graphical User Interface) will be described later in detail with reference to FIG.

Next, the evaluation area size determination unit 113 automatically determines the evaluation area size of the image (S4). A method for determining the evaluation area size of the image will be described later in detail with reference to FIGS.

Next, the feature amount extraction unit 114 extracts feature amounts from the labeling image using the determined evaluation area size (S5).

Next, the statistical analysis unit 115 statistically analyzes the relationship between the extracted feature amount and material properties (S6).

Next, the statistical analysis unit 115 prioritizes the effects of feature quantities on material properties (S7).

Next, the statistical analysis unit 115 proposes key features that need to be controlled (S8).

Next, the statistical analysis unit 115 determines a keyword and retrieves information on process control of key features (S9).

Next, the control unit 111 displays information on material process control (S10).

The user reviews the displayed information, further manufactures the material if necessary, and adds the data of the resulting material sample (S11).

<3. Work example 1 by GUI (specification of materials)>
FIG. 3 is an example of a GUI displayed on the image monitor 102 for specifying materials under the control of the control unit 111. As shown in FIG. A specific example of the processes S1 to S3 in FIG. 2 will be described with reference to FIG. In this example, material/process data 121, material property data 122, SEM image data 123, labeling image data 124, labeling model 125, feature amount data 126, statistical analysis result data 127, etc. are stored in the magnetic disk device 120 in advance. is stored.

In step S1, a "material name" is specified in the pull-down menu 301. FIG. Next, a "process No." is specified in the pull-down menu 302 from among a plurality of processes registered in advance for the material. In this example, "Material Name" is the major category and "Process No" is the minor category, but other classification methods may be used.

"Process No." specifies the combination of material and manufacturing method, and this information is stored in a database in the material/process data 121. Therefore, a sample can be obtained by specifying "material name" and "process No." can be specified. By operating the data acquisition button 303 with the keyboard or mouse 101 , desired sample data is acquired from the material/process data 121 of the magnetic disk device 120 . The operation of the GUI buttons is the same below.

In the example of FIG. 3, the composition of the material is displayed in the window 304 based on the acquired material/process data 121 of the sample. The material composition is known to the user who created the sample, and is stored in the material/process data 121 . You can refer to the periodic table if necessary.

A window 305 displays manufacturing conditions. The manufacturing conditions are known to the user who created the sample, and are stored in the material/process data 121 . For example, temperature, pressure, additives, stirring time, and the like. Also, if the material property data 122 of the sample has already been acquired in the magnetic disk device 120 , it is displayed in the window 306 . For example, lattice constant, mass density, Vickers strength, electrical conductivity, thermal conductivity and the like.

In step S2, by operating the observation image acquisition button 307, n SEM images of the sample are acquired from the SEM image data 123 of the magnetic disk device 120. FIG. (Part of) the acquired SEM image is displayed in window 308 . Assume that the SEM image (and labeling image) is standardized in advance with a predetermined size and resolution. As an example, a standardized size is stored in a database as the minimum image unit.

In process S3, a labeling image corresponding to the acquired SEM image is acquired. If the labeling image data 124 already exists in the magnetic disk device 120, the labeling image acquisition button 309 is operated to retrieve the data. (Part of) the called labeling image is displayed in window 308 .

In FIG. 3, three SEM images and three labeling images are displayed vertically, but the display method and the number of displayed images are arbitrary.

If the magnetic disk device 120 does not have the labeling image data 124 , the segmentation unit 112 operates the keyboard or the mouse 101 to visually label the SEM image on the window 308 . After labeling is completed, the button 310 is operated to confirm or determine the labeling information. Labeling information is, for example, the name and color of each phase (region). For example, "Phase A, RGB (255, 200, 100)" is determined. The labeling image for which labeling has been determined is displayed in window 308 and is recorded as labeling image data 124 in magnetic disk device 120 .

Further, as described above, when labeling is performed automatically by applying machine learning technology, the segmentation unit 112 calls the labeling model 125 by operating the labeling execution button 311 to automatically perform labeling. The labeling model 125 is, for example, a DNN (Deep Neural Network) learned by supervised learning using pairs of SEM images and labeling images labeled by the user as teacher images.

By operating the image data button 312, data such as the observation conditions of each SEM image can be called up, corrected, and confirmed. Data such as observation conditions include, for example, image name, observation device ID, operator, acceleration voltage, WD (working distance), magnification, scan time for image storage, etc. These are metadata of the SEM image data 123 .

After the above processing is performed and the user can grasp the content of the sample, the button 313 is operated to proceed to the evaluation area size determination processing S4.

<4. Evaluation Area Size Decision Processing>
FIG. 4 is a detailed flow of the evaluation area size determination process S4 executed by the evaluation area size determination unit 113. As shown in FIG. The SEM image data and labeling image data of the sample created by the material and process designated by the user are read into the working area of the semiconductor memory 110 in the processes from S1 to S3.

In process S4-1, the evaluation area size determination unit 113 determines a method of combining image data. The combining method refers to how many images are combined (or added). For example, combine 3 images horizontally, or combine 9 images 3×3, and so on. The combining method may be determined by default, may be specified by the user each time, or may be selected by the user from options prepared in advance. In the following example, an example of combining three images horizontally will be described.

Note that the image combination and division may be performed using the SEM image, or may be performed using the labeling image. Since both correspond, replacement is possible. In the following, the labeling image will be used for the sake of explanation.

In process S4-2, a plurality of images are combined by the determined method. Combining refers to stitching together multiple images.

In process S4-3, the combined image is equally divided into three size evaluation images (hereinafter sometimes referred to as "evaluation images"). The reason for dividing into three is to calculate these deviations later.

In process S4-4, the feature quantity extraction unit 114 is called to calculate the feature quantity of the labeling images corresponding to the three evaluation images. If the processing up to S4-3 is performed using a labeling image, the feature amount may be calculated from the labeling image. If the processing up to S4-3 is performed with the SEM image, the corresponding labeling image is read from the labeling image data 124. FIG.

The feature values of the labeling image are, for example, the number of phases, the area ratio of each phase, the shape information of each phase, the area of each phase (average value, maximum value, minimum value, variation), and the maximum linear length of each phase (average value, maximum value, minimum value, variation), etc., may be arbitrarily selected. In this example, the area ratio of each phase, which is considered to have a large effect on the material properties, will be described as an example.

After the feature amount is calculated, the evaluation area size determining unit 113 calculates the deviation of the feature amount. where Deviation is
Deviation =
(std(Total area of the feature))/(ave(Total area of the feature))
defined by std means the standard deviation, ave means the average value, and "Total area of the feature" means the whole area of interest.

In process S4-5, the calculated deviation is compared with a predetermined condition. Typical processing compares the deviation to a predetermined threshold to determine if it is less than the threshold. If it is equal to or greater than the threshold, the process proceeds to step S4-6, and if it is smaller than the threshold, the process proceeds to step S4-7.

In process S4-6, an image is added to the image combined in process S4-2 to combine a plurality of images. After that, the processes S4-3 to S4-5 are repeated.

In process S4-7, if the deviation is smaller than the threshold, the size of the evaluation image divided in process S4-3 is proposed to the user via an image monitor or the like as an evaluation area size.

In process S4-8, the next process proceeds according to the evaluation area size determined by the user.

FIG. 5 shows the concept of combining and dividing labeling images in processing S4-2 and processing S4-3. Three labeling images 501 whose sizes are standardized as shown in (A) are combined horizontally as shown in (B). After that, the evaluation image 502 is generated by equally dividing into three as shown in (C). In addition, although three labeling images 501 are combined in FIG. 5, the number may be two or four or more. Three evaluation images 502 are required to calculate the deviation, but the number does not have to be three or more if other evaluation methods, which will be described later, are used. In the example of FIG. 5, the evaluation area size ^pX2 of the evaluation image 502 is _W11 × _W12 .

The labeling images 501-1, 501-2, 501-3 may correspond to arbitrary regions on the sample and need not correspond to contiguous regions. For example, it may be randomly selected from SEM image data of the same sample or the same process stored in the magnetic disk device 120 . In the labeling image 501, labels such as phases A, B, and C are applied.

A method of determining the evaluation area size according to this embodiment will be described with reference to FIG. Here, the area ratios of a plurality of phases A, B, and C are used as the feature quantity of the labeling image. Phases A, B, and C may be distinguished by their elemental content or by other properties such as crystal structure.

In FIG. 6, the horizontal axis indicates the evaluation area size pX ² , and the vertical axis indicates the feature quantity deviation. The threshold used in S4-5 is indicated by dev _th . The threshold dev _th is determined in advance by the user and stored in the evaluation area size determining unit 113 .

As shown in FIG. 6, generally, as the evaluation area size increases, the feature amount is averaged, so the variation (deviation in this example) of the feature amount between the evaluation images decreases. In this embodiment, the evaluation area size A _det with all phase deviations smaller than the threshold dev _th is proposed to the user. That is, it is possible to propose the minimum evaluation area size that can adequately represent the characteristics of the sample. Note that the threshold may be determined for each phase.

FIG. 7 describes a method of determining the evaluation area size when the macro size is introduced. In FIG. 7, the horizontal axis indicates the evaluation area size pX ² , and the vertical axis indicates the feature amount deviation. In this example, it is assumed that even if the evaluation area size is increased, the deviation does not fall below the threshold dev _th . In this case, the macro size A _MC is defined together with the deviation threshold dev _th . If the evaluation area size A _det reaches the macro size A _MC before the deviation falls below the threshold dev _th , the macro size is set as the evaluation area size without further increasing the evaluation area size.

FIG. 8 shows the concept of additional processing of SEM image data in processing S4-6 when the calculated deviation is equal to or greater than the predetermined threshold. Compared with FIG. 5, six additional labeling images 501add are added to the labeling images 501-1, 501-2, and 501-3 by processing S4-6 as shown in (A). Combine this as shown in (B). After that, as shown in (C), the evaluation image 502add is generated by equally dividing into three. It should be noted that how images are added/combined is defined in step S4-1.

Among the three evaluation images 502 in FIG. 5, the evaluation image 502-2 in the middle does not include the characteristic phases B and C, and is not suitable for evaluating the characteristic amount of the material. In the evaluation image 502add of FIG. 8, the phases A, B, and C are included in each evaluation image substantially equally, and the feature amount of the material can be appropriately evaluated from the evaluation image. In particular, as shown in FIGS. 5 and 8, when phases B and C occupying a small area with respect to phase A occupying a large area (for example, 90% or more) affect the properties of the material. , it is important that these phases are included in the feature extraction region. In this embodiment, it is possible to automatically determine an appropriate feature amount extraction region size.

<5. Supplementary Explanation>
(1) In the above description, a plurality of images are combined in process S4-2, but if the size of the standardized image is large, process S4-2 may be omitted.
(2) In the above description, in process S4-3, the combined image is divided evenly into three evaluation images. When an evaluation index is used, it may be 2 or more. for example,
A) Variance δ2 of the feature amount of the evaluation image is smaller than the threshold.
B) All the differences between the feature amount of each evaluation image and the average value of the feature amounts are smaller than the threshold.
C) The standard deviation δ of the feature is smaller than the threshold.
d) All the differences between the feature amounts of the evaluation images are smaller than the threshold.
(3) In the above description, labeling images are combined and divided in steps S4-2 and S4-3, but corresponding SEM images may be combined and divided.
(4) In the above description, the feature amount of the labeling image is calculated in step S4-4, but the feature amount of the SEM image may be used instead. SEM image features include image frequency, average brightness, average contrast, and the like. However, it is preferable to use the labeling image because the labeling image directly represents the feature quantity of the sample to be analyzed.

<6. Work Example 2 Using GUI (Determination of Evaluation Area Size)>
FIG. 9 is an example of a GUI displayed on the image monitor 102 under the control of the control unit 111. As shown in FIG. By operating the button 313 in FIG. 3, the processes of S4-1 to S4-6 described above are performed, and then the screen in FIG. 9 is displayed to determine the evaluation area size.

A specific example of the processes S4-7 to S4-8 in FIG. 4 will be described with reference to FIG. The recommended evaluation area size 901 obtained by the processes S4-1 to S4-6 in FIG. , is displayed in window 904 . In this example, the recommended evaluation area size 901 and the like are displayed for multiple samples, so the processes in FIGS. 2 and 4 are also performed for multiple samples.

In the system, the size of image data is handled in units of pixels, but it is preferable to convert it to actual size data and display it. Here, the recommended evaluation area size 901 is calculated from pX ² =W ₁₁ ×W ₁₂ (see FIGS. 5 and 8). A default value of the evaluation area size 905 may be the same as the recommended evaluation area size 901 . The user changes the evaluation area size 905 as required.

A warning is displayed in window 906 if the evaluation area size 905 is not optimized. In this example, the evaluation area size of process number 1 is the size that the deviation does not satisfy the threshold in the determination of process S4-5 in FIG. In this case, the user can select whether to continue by ignoring with the button 907 or to add more images and increase the size with the button 908 . If the size has been increased, the process returns to S4-6 in FIG.

The recommended evaluation area size 901 is proposed in process S4-7 when the deviation does not satisfy the threshold in the process S4-5 of FIG. 4 because the macro size A _MC is defined as described above. is the case. A button 908 allows the user to ignore the macro size A _MC and increase the size. Alternatively, the user can enter an evaluation area size 905 that is smaller than the recommended evaluation area size 901 . Again, a warning is displayed in window 906, but the user can ignore it with button 907 and continue.

A window 904 also shows feature values 909 calculated from the labeling image of each sample. For these feature amounts, the feature amounts calculated in process S4-4 of FIG. 4 may be used. In this example, the area ratios of the phases A, B, and C, the average size of each phase, and the like are shown as the feature quantity 909 . In this system, the area ratio of the phase is represented by "% (name of phase)". When the evaluation area size 905 is changed, the user can use the reanalyze button 910 to recalculate the feature amount.

When the evaluation area size 905 is finally determined, the user presses the feature amount extraction button 911 to start the actual feature amount extraction processing (S5 in FIG. 2). In the actual feature amount extraction process S5, it is desirable to extract the feature amount in more detail and multifacetedly than the feature amount calculated in the process S4-4 of FIG. The extracted feature amount can be displayed in window 904 . In this embodiment, the feature amount extraction unit 114 stores algorithms for extracting various feature amounts, and the control unit 111 calls them. The extracted feature amount is stored in the magnetic disk device 120 as the feature amount data 126 .

Note that the check button 912 can apply an evaluation area size based on typical conditions. In the example of FIG. 9, samples with multiple process numbers are displayed, and the evaluation area size is obtained for each sample. However, the user can use the check button 912 to apply the evaluation area size of one sample to other samples if it is known in advance that the process conditions of the samples are similar. For example, when there is little variation in the texture within the range of process conditions, select a representative condition (e.g. Process No. 1) from among the process conditions, determine the image evaluation size, and then use the same image evaluation size for all process conditions. conduct.

When the feature amount extraction process S5 is completed, the feature amount effectiveness evaluation button 913 can be used to start the feature amount effectiveness extraction process (S6 to S10 in FIG. 2). In the feature quantity effectiveness extraction processing, a correlation analysis is performed between the feature quantity and the material properties. Correlation analysis is performed by a correlation calculation unit provided as part of the algorithm of the feature quantity extraction unit 114 . As a result of performing the correlation analysis, it is possible to know which feature value has the greatest influence on the material properties. That is, it is possible to specify the feature quantity important for controlling the characteristic. Next, search for what kind of processes and materials are available to control the feature quantity. By examining the search results, the user can plan the next experiment and obtain new samples.

<7. Analysis of relationship between feature quantity and material properties>
As a result of processing S5, the feature amount data 126 could be prepared for each sample. The feature quantity data 126 can be cross-referenced with the material/process data 121, the material property data 122, the SEM image data 123, and the labeling image data 124. For example, using the process number or sample code as a key, it corresponds to a specific sample. You can identify the data that Therefore, for example, it is possible to draw a graph in which the horizontal axis represents the feature amount and the vertical axis represents the material properties.

Referring to FIG. 2 again, the statistical analysis unit 115 analyzes the relationship between the feature quantity and the material properties based on the above data in process S6. The method of analysis can use known principal component analysis and linear analysis, and is not particularly limited. Furthermore, the statistical analysis unit 115 prioritizes the effect (correlation) of the feature quantity on the material properties in processing S7. For example, the order of correlation coefficients is given higher priority. Then, in step S8, the statistical analysis unit 115 proposes the high-priority feature amount as an important feature amount that becomes a key to be controlled.

The results of the statistical analysis are recorded in the magnetic disk device 120 as statistical analysis result data 127, in which correlations (for example, correlation coefficients) of feature quantities with respect to each material characteristic are recorded.

FIG. 10 is an example of a GUI showing analysis results of the processes S6 to S8 displayed on the image monitor 102 under the control of the control unit 111. FIG. A pull-down menu 1001 allows the user to select the material properties to be analyzed. Vickers intensity is selected here. As a result, on the left side of the window 1002, a graph showing the correlation between the Vickers intensity and the feature amount is displayed in descending order of the degree of correlation. On the right side of the window 1002, as more detailed information, a graph showing the relationship between the Vickers intensity and the feature amount for the top three feature amounts with the highest degree of correlation is displayed. For example, these three feature amounts are proposed as key feature amount candidates that need to be controlled. The user can change the number of features to be displayed using a pull-down menu 1003 .

As described above, the user can know what feature amount is estimated to be important for controlling the Vickers intensity. Results such as the proposed feature amount are stored as statistical analysis result data 127 in the magnetic disk device 120 by pressing the record button 1004 . Next, the user uses the process information search button 1005 to start the process information search process S9 and the result display process S10.

Note that the sample addition button 1106 is for recalculating the relationship between the feature amount and the material properties when new sample data is input.

<8. Process Information Search Processing>
FIG. 11 is a flowchart showing the details of the process information search processes S9 and S10. These processes are executed by the process information search unit 116 .
FIG. 12 is an example of a GUI displayed on the image monitor 102 under the control of the control unit 111 in the process information search process S9. Description will be made with reference to FIGS. 11 and 12. FIG.

In FIG. 12, the user inputs the material properties desired to be improved, the purpose, or the content desired to be controlled (“increase”, “decrease”, etc.) in a window 1201 . The content entered by the add purpose button 1202 is determined as the goal (S9-1).
The process information search unit 116 determines keywords by, for example, known syntax analysis based on the words input in step S9-1. "Vickers" and "strength" are displayed as keywords in the window 1203 of FIG. 12 (S9-2). These keywords are defined in the system of the embodiment, and the variable names can be used as they are.

Based on the keywords "Vickers" and "strength," the process information retrieval unit 116 searches the statistical analysis result data 127 of the magnetic disk device 120 to find the correlation between the feature quantity important for controlling the "Vickers intensity" and the Vickers intensity. data. As a specific example, for example, data indicating the relationship between the top nine feature amounts extracted as feature amounts highly related to the Vickers strength in FIG. 10 and the "Vickers strength" is called. Then, using the feature amount, a characteristic prediction model is generated, and the model with the highest accuracy is selected.

From the statistical analysis result data 127, it is possible to identify a feature amount highly correlated with a predetermined characteristic, but it is not known what kind of effect the feature amount has on the characteristic. Therefore, a characteristic prediction model is created that indicates the relationship between the feature amount and the characteristic. The characteristic prediction model is, for example, a linear regression model, and a well-known method is used to create a model that clearly shows the relationship between the characteristic amount and the characteristic using the characteristic amount as the explanatory variable and the characteristic as the objective variable.

For this purpose, for example, model candidates are created by using highly correlated feature quantities one by one as explanatory variables. When the feature quantity is nine in FIG. 10, nine model candidates are created. Alternatively, model candidates are created using multiple feature values as explanatory variables. After creating a plurality of model candidates, the model that most accurately expresses the characteristics is selected (S9-3).

FIG. 13 shows the concept of evaluating the relationship between experimental properties obtained from experiments and properties obtained from models (predicted properties) for three models. Suppose that X1, X2, and X3 are highly related feature values, and model 1 uses only X1, model 2 uses X1 and X2, and model 3 uses X1, X2, and X3 to represent characteristic Y. . In this example, model number 3 is selected as the property prediction model because it has the smallest error and best represents the experimental properties.

Next, the process information search unit 116 determines the feature amount used in the model selected in step S9-3 as the key feature amount. In the case of FIG. 13, X1, X2 and X3 are determined as key feature amounts (S9-4).

Next, the process information search unit 116 determines a keyword based on the feature quantity determined in step S9-4 (S9-5). For example, let the feature quantities X1, X2, and X3 be "%phase 1", "Average size of phase 1", and "%phase 2" shown in FIG. These data expressions are defined in the system of the example, "%phase X" indicates the area fraction of the phase consisting of element X, and "Average size of phase X" indicates the area of the phase consisting of element X. Indicates the average size (eg area). The process information search unit 116 generates keywords from these feature amounts. For example, "%phase 1", "%phase 2", "average size of phase 1". 1 and 2 of phase 1 and phase 2 indicate some element names (element 1 and element 2). Note that the sample material name "ABC" was previously extracted as a keyword (S9-2).

After that, the relationship between the process and the feature quantity is searched, and the process branches depending on whether the system has the relationship data between the process and the feature quantity (S9-6). When searching from the relationship between the process conditions and the feature values, the data in the magnetic disk device 120 is used for searching, and the relationship between the process conditions and the feature values is displayed. Also, when searching from external information, related information (for example, literature) is displayed.

<8-1. If the system has process information>
In the material/process data 121 and the material property data 122 stored in the magnetic disk device 120, when the feature amount, material/process, and property of the sample are associated, a key is used to obtain the predetermined property. (S9-7A). For this purpose, in the characteristic prediction model selected in (S9-3), the change required in the feature amount is determined in order to obtain the characteristic change specified in the window 1201 of FIG. For example, in order to increase the characteristic Y in the model of number 3 in FIG. 13, the feature quantities X1 and X3 are increased and X2 is decreased.

In process S9-8A, a keyword is determined based on the change in feature quantity determined in S9-7A. For example, "characteristic amount X1*increase", "feature amount X2*decrease", and "feature amount X3*increase". By operating the search button 1204, the process information search unit 116 searches the material/process data 121 and the feature amount data 126 using the keywords determined in steps S9-2, S9-5, and S9-8A.

For example, in the material/process data 121, if the relationship between a sample of a specific material and the process can be identified, and in the feature data 126, if the relationship between the sample of a specific material and the feature quantity can be identified, the relationship between the process and the feature quantity can be specified. Data showing changes can be generated. Such data may be generated from the material/process data 121 and the feature quantity data 126 by batch processing and stored in the magnetic disk device 120 as the process/feature quantity relationship data 128 . At this time, it is also desirable to name the data, for example, "relational data between feature quantity X1 and process P1" and attach tags such as "increase" and "decrease".

Such process/feature quantity relational data 128 is searched using the above-described keyword.

A window 1205 in FIG. 12 displays a graph showing the relationship between the feature amount extracted as a result of the search and the process, together with arrows indicating "increase" and "decrease" of the feature amount. In this example, the horizontal axis indicates the number of times the material is stirred, and the vertical axis indicates various feature amounts. This allows the user to know which process to modify to produce a given property change in the material.

<8-2. If the system does not have process information >
If the system does not have related data on processes and feature quantities, it searches for process information using an existing database or the like.

If there is no data indicating the relationship between material properties and processes, determine general expressions for keywords (S9-7B). As a method of generalization, for example, a thesaurus is used to change the keyword "ABC" determined in step S9-2 to "alloy" or "polymer". Or change from "Vickers strength" to a broader keyword "strength" or "hardness". Also, the keywords determined in step S9-5, such as "%phase 1", are changed to "element 1" and "area ratio". Since the keywords used in window 1203 may use expressions unique to this system, window 1206 shows corresponding general keywords. Keyword conversion can be performed using a correspondence table or dictionary. For example, "%phase1" is not a general notation, so it is converted to "element 1" and "area ratio".

Further, in the process S9-7B, the user may be allowed to edit or add keywords on the screen shown in FIG. The final determination of keywords may be made after confirmation by the user. Alternatively, user confirmation may be omitted.

Then, it connects to an existing general search engine or research database through the input/output interface 104 (S9-8B), searches for information on process control of microstructure features, and displays it in a window 1207 (S9-10B). ). When the sample data increases, the sample addition button 1208 is used to perform recalculation.

According to the embodiment described above, in material development, the evaluation area size of each sample is determined using a tissue measurement device or a tissue image, and the tissue feature amount is extracted within the evaluation size, and the material characteristics are most suitable. It is possible to provide a material structure analysis system that analyzes a highly influential structure feature quantity and provides information on control of the structure feature quantity. By determining the evaluation area size of the tissue image, a highly reliable feature amount can be extracted even when the amount of heterogeneity is small. In addition, information can be provided regarding tissue control that is important for a given characteristic. In addition, when image data is input and machine learning is performed on a model to obtain a predetermined output, by using the area of the determined evaluation area size as the area of the input image data, it is possible to obtain an appropriate output for the model. value can be obtained.

Control unit 111, segmentation unit 112, evaluation area size determination unit 113, feature amount extraction unit 114, statistical analysis unit 115, process information search unit 116, material/process data 121, material property data 122, SEM image data 123, labeling Image data 124 , labeling model 125 , feature amount data 126 , statistical analysis result data 127 , process/feature amount relationship data 128 .

Claims (14)

An information processing system that includes an input unit, an output unit, a storage unit, and a processing unit, and calculates a feature amount from a labeling image of an observed image obtained by observing a sample to evaluate the sample,
a feature amount extraction unit that calculates a feature amount of each image from the plurality of observation images or the plurality of labeling images;
An evaluation area size determination unit that calculates the feature amount based on the distribution of the feature amount of each image and determines the area of the labeling image for evaluating the sample,
The evaluation area size determination unit
When evaluating labeling images of observed images obtained by observing samples with different manufacturing processes, and when the difference in the manufacturing process between the samples is within a predetermined range,
Applying the area of the labeling image for assessing the sample from one manufacturing process to the labeling image for assessing the sample from the other manufacturing process ,
Information processing system. An information processing system that includes an input unit, an output unit, a storage unit, and a processing unit, and calculates a feature amount from a labeling image of an observed image obtained by observing a sample to evaluate the sample,
a feature amount extraction unit that calculates a feature amount of each image from the plurality of observation images or the plurality of labeling images;
An evaluation area size determination unit that calculates the feature amount based on the distribution of the feature amount of each image and determines the area of the labeling image for evaluating the sample,
having a correlation calculation unit,
The correlation calculation unit
The input is the feature value of the labeling image of the observed image of the sample observed by different manufacturing processes,
taking as input the material properties of the samples from the different manufacturing processes;
calculating the degree of influence of the feature quantity on the material properties ;
Information processing system. has a process information search unit,
The process information search unit is
inputting a key feature amount selected based on the degree of influence on the material properties, and determining a keyword for searching based on the key feature amount;
3. The information processing system according to claim 2 . The storage unit
Storing process/feature data that records the relationship between the process conditions of the sample by different manufacturing processes and the feature of the labeling image of the observation image obtained by observing the sample,
searching the process/feature data with the keyword;
4. The information processing system according to claim 3 . The evaluation area size determination unit
Determining the area of the labeling image based on at least one of the deviation of the feature amount, the variance of the feature amount, the difference between each feature amount and the average value of the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts. ,
3. The information processing system according to claim 1 or 2 . The evaluation area size determination unit
until at least one of the deviation of the feature amount, the variance of the feature amount, the difference between each feature amount and the average value of the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts becomes smaller than a predetermined threshold, increase the area
3. The information processing system according to claim 1 or 2. The labeling image is an image in which each portion in the image of the observation image is divided into a plurality of phases based on at least one of the contained elements and the crystal structure.
3. The information processing system according to claim 1 or 2 . The feature amount extraction unit calculates at least one of the area ratio, average area, and shape information of each phase of the labeling image as a feature amount.
The information processing system according to claim 7 . The feature amount extraction unit calculates the feature amount of each image from a plurality of the observed images or a plurality of the labeling images obtained from samples of the same process.
3. The information processing system according to claim 1 or 2 . The feature quantity extraction unit is
Applying the area of the labeled image determined by the evaluation area size determination unit, calculating the feature amount from the labeled image of the observed image obtained by observing a plurality of samples;
The information processing system according to claim 1 or 2 . An information processing method that is executed by an information processing device that includes an input unit, an output unit, a storage unit, and a processing unit, and calculates a feature amount from a labeling image of an observation image obtained by observing a sample,
feature quantity extraction processing for calculating the feature quantity of each image from a plurality of the observed images or a plurality of the labeling images relating to samples created under the same conditions;
Evaluation area size determination processing for determining the area of a labeling image for evaluating the sample by calculating the feature amount based on the distribution of the feature amount of each image;
and
In the evaluation area size determination process,
A first step of preparing a labeling image that has been standardized in advance with a predetermined size and resolution;
a second step of combining a plurality of said normalized labeling images;
a third step, subsequent to the second step, of dividing the combined labeling images into a plurality of size evaluation images;
a fourth step of calculating a feature amount for each of the size evaluation images, following the third step;
Following the fourth step, a fifth step of evaluating the state of variation in the feature amount,
Following the fifth step, based on the variation situation, return to the second step to increase the number of normalized labeling images to be combined, or depending on the size of the current size evaluation image, a sixth step of determining whether to determine the area of the labeling image to assess the sample;
and
A seventh step of applying the area of the labeling image determined in the sixth step to calculate the feature amount from the labeling image of the observation image obtained by observing a plurality of samples imaged under conditions with different manufacturing processes or compositions. ,
an eighth step of preparing material property data for a plurality of samples imaged under different conditions of the manufacturing process or composition;
A ninth step of performing a correlation analysis between the feature quantity and the material properties, and evaluating the degree of influence of the feature quantity on the material properties;
A tenth step of determining a keyword for data search based on the feature amount adapted to the target value of the material property, based on the evaluation result of the degree of influence of the feature amount;
information processing method for In the fifth step,
Calculate the feature deviation, the feature variance, the difference between each feature and the mean of the feature, the standard deviation of the feature, and the difference between the features;
The information processing method according to claim 11 . In the sixth step,
If the deviation of the feature amount, the variance of the feature amount, the difference between each feature amount and the average value of the feature amount, the standard deviation of the feature amount, and the difference between the feature amounts are smaller than a predetermined threshold, the current size evaluation image determining the area of the labeling image for evaluating said sample by the size of
The information processing method according to claim 11 . When machine learning a model that takes image data as input and obtains a predetermined output,
using the determined area of the labeling image for evaluating the sample as the area of the input image data;
The information processing method according to claim 11 , wherein

Images