JP2022136745A - Analysis device, analysis method and program - Google Patents

Abstract

To provide an analysis device, an analysis method, and a program for improving availability of characteristic estimation of a microstructure of a steel material using map data.SOLUTION: An analysis device includes: a data acquisition unit that acquires map data representing a microstructure of a steel material; and a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an analysis device, analysis method and program.

Alloying elements such as carbon, manganese and silicon are actively used in the production and development of steel materials. In particular, carbon generates compounds such as cementite and develops structures such as bainite and pearlite by adjusting the amount of addition and undergoing heat treatment. Moreover, manganese has a function of stabilizing the austenite structure. In addition, silicon has a function of promoting the formation of ferrite structure and a function of improving hardenability. Morphological aspects such as the phase fraction and grain shape of these structures affect the mechanical and chemical properties of the final product. Additionally, structures such as chromium, nickel, niobium, and boron may be added to control desired properties. Also in the heat treatment, the properties are controlled by single or multiple treatments such as hot rolling, cold rolling, annealing, quenching, and tempering. As a result, the final steel material contains martensite, bainite, ferrite, pearlite, austenite, other precipitates, etc., and exhibits a wide variety of microstructures and their distributions. These analyzes (tissue identification) include optical microscopy (OM), scanning electron microscopy (SEM), electron backscatter diffraction (EBSD) techniques, electron probe microanalyzer (EPMA), transmission electron microscopy (TEM), etc. Making full use of various analytical instruments and utilizing the map data, we perform visual structure estimation, quantification of alloy composition from concentration distribution, local orientation analysis, etc.

Among them, a technique for estimating the properties of steel materials based on images of microstructures has been proposed. For example, Patent Literature 1 describes a technique of estimating the material properties of a material using a neural network model based on a scanning electron microscope image of the material, and the estimation result includes estimation confidence information. Further, in Patent Document 2, image data of sparks generated when steel is ground is used as input data, component information of the steel is used as teacher data, and a trained model for estimating the components of the steel is machined from the image data. Techniques that build by learning are described.

JP 2019-12037 A JP 2020-9435 A

According to the technology for estimating material properties using image processing and machine learning as described above, it is possible to quantify information such as structures, precipitates, and inclusions without relying on conventional visual judgment by experts. can be done. However, since the above technology assumes images captured by the same method as input data, it is possible to adapt to changes in imaging conditions and to select the type of image suitable for identification according to the tissue. It was difficult.

Therefore, it is an object of the present invention to provide an analysis apparatus, an analysis method, and a program that can secure the versatility and quantification of the characteristic estimation of the microstructure of steel materials using map data and improve the availability. do.

According to one aspect of the present invention, a data acquisition unit that acquires map data expressing the microstructure of a steel material; and a learning model constructed according to tag information associated with the map data. and a segmentation unit for classifying elements into segments.

According to another aspect of the present invention, acquiring map data expressing the microstructure of a steel material; into segments.

According to still another aspect of the present invention, a data acquisition unit that acquires map data expressing the microstructure of a steel material; A program is provided for causing a computer to function as an analysis device comprising a segmentation unit that classifies elements of data into segments.

According to the above configuration, tag information is associated with map data, and segmentation is performed using a learning model built according to the tag information. Segmentation can be performed using the model, ensuring versatility and quantitativeness in estimating the characteristics of the microstructure of steel materials, and improving availability.

1 is a schematic block diagram of an analysis device according to one embodiment of the present invention; FIG. 2 is a diagram showing an example of tag information associated with map data in the example of FIG. 1; FIG. 6 is a flow chart showing an example of a process of updating a model in the analysis device according to one embodiment of the present invention; 4 is a flow chart showing an example of processing for correcting input map data in the analysis device according to one embodiment of the present invention.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

FIG. 1 is a schematic block diagram of an analysis device according to one embodiment of the present invention. As illustrated, the analysis device 100 includes a data acquisition unit 101, a segmentation unit 102, a model construction unit 103, a post-processing unit 104, a result output unit 105, a reliability calculation unit 106, and a model update unit. 107 , a data correction instructing unit 108 , a data correcting unit 109 , and a characteristic prediction unit 110 . The analysis device 100 is a computer including a processor such as a CPU (Central Processing Unit), a storage device, a communication device, an input/output means, etc. The above functional portions are realized by the processor operating according to a program. The program is stored in a storage device, or stored in a removable storage medium and read into analysis device 100 . In the database of the analysis device 100, image data or local analysis data of the steel microstructure is stored as map data 121, learning model algorithms and parameters are stored as model data 122, and classification by the segmentation unit 102 described later. The result (including the classification result after post-processing by the post-processing unit 104) is stored as the result data 123. FIG.

The analysis apparatus 100 as described above may be implemented by, for example, a single computer, or may be implemented by being distributed among a plurality of computers. At least part of the functional parts of analysis device 100 may be implemented in one or a plurality of server devices, and processing may be executed by transmitting data or processing requests from client devices. Each functional part will be further described below.

The data acquisition unit 101 acquires image data or local analysis data, which are examples of map data 121 representing the microstructure of the steel material. The data acquired may be image data, such as optical microscopy (OM) images, scanning electron microscopy (SEM) images or transmission electron microscopy (TEM) images, or electron probe microanalyzer (EPMA) or electron backscatter diffraction. It contains local analysis data acquired with the (EBSD) method. Such data are common in that one or a plurality of values such as brightness, crystal orientation, and component composition are mapped to elements having position information within a predetermined region. can be treated in the same way in the analysis by Such data is also referred to as map data in this specification. In the following description, a set of combinations of elements and values of the map data 121 may be referred to as an image, but this expression does not limit the map data 121 to images as described above.

In this embodiment, the map data 121 acquired by the data acquisition unit 101 is associated with tag information. In the case of steel material image data or local analysis data, a data acquisition technique such as OM, SEM, EPMA or EBSD is associated with each piece of map data 121 as tag information. In addition, the steel type classification (thin plate, thick plate, steel pipe, etc.), composition, heat treatment, etching conditions, or imaging conditions for image data or local analysis data (type of SEM radiation source, etc.) are used as tag information in individual map data. 121 may be associated. Further, by combining different types of map data based on the similarity of tag information, map data having multiple channels, such as RGB images, may be newly created from a database and used for learning. Tag information may also include information that can be extracted upon acquisition of image data or local analysis data. Specifically, for example, EPMA data can extract information on the chemical composition of steel materials at the time of acquisition, EBSD can extract phase fractions such as ferrite and austenite, average grain size, etc., and SEM and OM can extract information at the time of acquisition. A standard amount of carbon content in steel can be extracted.

FIG. 2 is a diagram showing an example of tag information associated with map data in the example of FIG. The tag information associated with the image file (map data) includes the type of measuring equipment, measurement conditions, composition of each chemical component of steel, type of each structure, fraction of each structure, grain size (average, distribution), Items appropriately selected from items such as the history of heat treatment temperature and heat treatment rate are included (preferably multiple items are included). For example, the tag information includes at least one of the steel grade classification, composition, heat treatment, etching conditions, or imaging conditions for image data or local analysis data. In the example shown in FIG. 2, items such as measuring equipment (SEM), ferrite fraction indicating composition, austenite fraction, ferrite average grain size, C composition, Mn composition, Si composition, annealing temperature and cooling rate are included. . The item of tag information may be, for example, recorded at the time of manufacture or obtained by separate measurement or analysis, or may be extracted at the time of obtaining image data or local analysis data as described above. good. Note that items to be included in the tag information may be changed, such as by addition or deletion, as appropriate when the segmentation described later does not produce good results (such as when YES is not obtained in step S104 shown in FIG. 3).

The segmentation unit 102 classifies the elements of the map data 121 acquired by the data acquisition unit 101 into segments using a learning model. This process is also called segmentation in this specification. Here, since the map data 121 is data in which values are mapped to elements having positional information, as described above, segmentation is performed on values mapped to individual elements and values between elements having a predetermined positional relationship. It is the process of classifying elements into segments based on their relationships. Learning models include, for example, S. M. Azimi, D. Britz, M. Engstler, M. Fritz and F. Mucklich, "Advanced Steel Microstructural Classification by Deep Learning Methods," Scientific Reports, 8 (2018), 2128. FCNN (Fully Convolutional Neural Network) system model, W. Wang, et al., "Learn to segment single cells with deep distance estimator and deep cell detector," Computers in Biology and Medicine, Vol. 108, May 2019, Algorithms such as the CNN hybrid model as described in pp. 133-141 are used. In this embodiment, the classification result by the segmentation unit 102 is given as the probability that each element is classified into a segment calculated by the learning model.

The model construction unit 103 constructs a learning model used by the segmentation unit 102 according to tag information associated with the map data 121 to be segmented. The model construction unit 103 is a learning model algorithm set in advance corresponding to a data acquisition method (for example, OM, SEM, EPMA, or EBSD) indicated by tag information of image data or local analysis data to be segmented, for example. may be selected. Alternatively, the model building unit 103 uses a learning model algorithm that has been shown to have a high degree of reliability, which will be described later, in past analyses, regarding a method of acquiring data indicated by tag information of image data or local analysis data to be segmented. may be selected. The model construction unit 103 constructs a model by reading algorithms and parameters from the model data 122 . As will be described later, the model construction unit 103 may perform additional learning or re-learning of the learning model by reading accumulated image data or local analysis data as teacher data.

The post-processing unit 104 executes post-processing for outputting the classification result by the segmentation unit 102 . Specifically, for example, the post-processing unit 104 executes image labeling and expansion/contraction processing using the OpenCV library. The post-processing unit 104 may perform knowledge-based correction of the image, for example. For example, if there is a segment with a size equal to or smaller than a threshold surrounded by other segments, the segment may be filled with the other segments. Also, the post-processing unit 104 may remove noise contained in the image according to a predetermined condition or pattern. Note that the classification result by the segmentation unit 102 can be output as image data even when the input is local analysis data.

The result output unit 105 outputs the classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104). Specifically, since the classification result by the segmentation unit 102 is output as an image as described above, the result output unit 105 outputs the image to a display, stores the image data in a storage device or a removable storage medium, for example. You can In addition, the result output unit 105 outputs the result of classification by the segmentation unit 102 to the database as result data 123, and based on the accumulated result data 123, the property prediction unit 110 executes the microstructure characteristics of the steel material. may be output.

The reliability calculation unit 106 calculates the reliability of the classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104). As described above, the classification result by the segmentation unit 102 is given as the probability that each element is classified into a segment. Therefore, even among the elements classified into the same segment, between elements with a higher probability (for example, a probability close to 100%) and elements with a low probability (for example, a probability close to 50%) , the reliability of the classification results is different. For example, the reliability calculation unit 106 may identify, as a low-reliability area, an area composed of elements whose classified segment probability is lower than a threshold.

The model update unit 107 updates the learning model constructed by the model construction unit 103 according to the reliability calculated by the reliability calculation unit 106 . For example, the model updating unit 107 may update the model when the area ratio of regions with low reliability in the classification result by the segmentation unit 102 exceeds a threshold. Here, the area ratio refers to the number of elements (for example, pixels number). Specifically, the model updating unit 107 uses the image data or the local analysis data extracted from the database based on the tag information of the image data or the local analysis data to be segmented as teacher data, and the model construction unit 103 may update the learning model by causing the to perform additional training or relearning of the learning model. The teacher data to be extracted is, for example, map data having common or similar tag information with image data or local analysis data to be segmented. Specifically, in the image data or local analysis data of steel materials, the steel type classification (thin plate, thick plate, steel pipe, etc.), composition, heat treatment, etching conditions, imaging conditions (type of SEM radiation source, etc.) are common or similar are extracted as teacher data. Therefore, the model updating unit 107 extracts tag information common or similar to tag information associated with image data or local analysis data (map data) to be segmented from the database, and extracts tag information associated with the extracted tag information. Perform additional training or retraining of the learning model using the map data obtained. For example, the method described in A. Seko, H. Hayashi, I. Tanaka, "Compositional descriptor-based recommender system for the materials discovery" J. Chem. Phys. Vol. 148 No. 24 (2018), 241719, Data similar to image data or local analysis data to be segmented may be extracted as training data by using indices such as Euclidean distance and cosine similarity.

Alternatively, the model update unit 107 may change the algorithm or network structure of the learning model constructed by the model construction unit 103 based on the tag information of the image data or local analysis data to be segmented. In this case, the model construction unit 103 may read the parameters already stored as the model data 122 for the algorithm after the change, or extract new teacher data by the above-described method and execute additional learning or re-learning. may For example, the model updating unit 107 may change the algorithm or network structure of the learning model if the reliability is not improved even after extracting teacher data and executing additional learning or re-learning of the learning model.

The data correction instruction unit 108 instructs correction of the map data 121 acquired by the data acquisition unit 101 according to the reliability calculated by the reliability calculation unit 106 . For example, when the area ratio of the low-confidence region in the classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104) exceeds a threshold, A data correction unit 109 to be described later may be instructed to correct the map data 121 . Here, the update of the learning model by the model update unit 107 and the correction of the map data 121 by the data correction instruction unit 108 may be executed according to a predetermined priority, for example. Specifically, correction of the map data 121 may be instructed when the reliability is not improved even if the learning model is corrected. Alternatively, the model updating unit 107 updates the learning model when regions with low reliability are dispersed in the image, and the data correction instruction unit 108 updates the map data 121 when regions with low reliability are concentrated. may be instructed to correct.

Data correction unit 109 corrects map data 121 acquired by data acquisition unit 101 in accordance with instructions from data correction instruction unit 108 . The modification of the map data 121 is performed, for example, by presenting operational guidelines for modifying image data or local analysis data to be segmented, and manually modifying the image data or local analysis data by the user according to the guidelines. good too. Alternatively, map data 121 may be modified according to A. Paul, A. Gangopadhyay, A. R. Chintha, D. P. Mukherjee, P. Das and S. Kundu, "Calculation of phase fraction in steel microstructure images using random forest classifier," IET Image Processing, Vol. 12, No. 8, August 2018, pp. 1370-1377 and S. Banerjee et al., "Segmentation of three phase micrograph: an automated approach," CUBE '12: Proceedings of the CUBE International Information Technology Conference, September 2012, pp. 1-4.

The property prediction unit 110 uses the results output to the result output unit 105 to predict the properties of the microstructure of the steel material. Specifically, when a new classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104) is acquired, the characteristic prediction unit 110 uses the new classification result and the result data 123 already stored in the database, the characteristics of the microstructure of the steel represented by the new classification result are predicted. The degree of similarity of the result data 123 may be determined by, for example, whether the tag information is common or similar to the image data or local analysis data to be segmented. Specifically, result data 123 having a common or similar steel type classification (thin plate, thick plate, steel pipe, etc.), composition, heat treatment, etching conditions, and imaging conditions (type of SEM radiation source, etc.) has a high degree of similarity. may be extracted as Further, search for result data 123 whose data itself is similar to the image data or local analysis data to be segmented using the method described in the literature by A. Seko et al., Euclidean distance, cosine similarity, etc. Then, the characteristics of the microstructure of the steel material may be predicted from the retrieved result data.

In the above case, the image data or local analysis data of the steel material may be associated with mechanical properties such as tensile strength, hole expandability, elongation, toughness, and hardness as tag information. In this case, the property prediction unit 110 calculates the degree of similarity between the new classification result by the segmentation unit 102 and the classification result stored in the result data 123, and the mechanical property associated with the classification result stored in the result data 123. Based on this, the mechanical properties of the microstructure of the steel material related to the new classification result may be predicted. Mechanical properties may be predicted as continuous values or as discrete categories. Models for predicting characteristics include, for example, SVM (Support Vector Machine), RF (Random Forest) or MLP (Multi-Layer Perceptron ), or CNN (Convolutional Neural Network) in the Keras library (https://keras.io/).

FIG. 3 is a flow chart showing an example of processing for updating a model in the analysis device according to one embodiment of the present invention. In the illustrated example, first, the data acquisition unit 101 acquires image data or local analysis data of the steel microstructure (step S101), and the segmentation unit 102 performs segmentation using the learning model (step S102). Furthermore, the reliability calculation unit 106 calculates the reliability of the classification result by the segmentation unit 102 (step S103). If the reliability is less than the threshold (NO in step S104), the model updating unit 107 updates the learning model (step S105), and segmentation is performed again on the same image data or local analysis data using the updated learning model. is executed (step S102). The reliability is calculated again (step S103), and if the reliability is equal to or greater than the threshold (YES in step S104), the classification result by the segmentation unit 102 is output (step S106). At this time, the property prediction unit 110 may search for existing result data 123 similar to the new classification result (step S107), and predict the properties of the microstructure of the steel material from the searched result data (step S108). .

FIG. 4 is a flowchart showing an example of processing for correcting input map data in the analysis device according to one embodiment of the present invention. In the illustrated example, the process proceeds in the same manner as in the example of FIG. 3 above, and if the reliability is less than the threshold (NO in step S104), the data correction instructing unit 108 instructs correction of the image data or the local analysis data. The correction unit 109 is instructed (step S111). When the input image data or local analysis data is corrected according to the instruction (step S112), the data acquisition unit 101 acquires the corrected image data or local analysis data (step S101), and the same process is executed thereafter. be.

<Example>
Table 1 shows the results of tensile strength property prediction performed by an analysis apparatus according to an embodiment of the present invention. For tensile strength property prediction, tag information with steel carbon, manganese, silicon composition, ferrite fraction, martensite and tempered martensite fraction, pearlite fraction, austenite fraction, and residual structure fraction was used. The ferrite fraction, martensite and tempered martensite fraction, pearlite fraction, austenite fraction, and residual structure fraction are phase fractions calculated based on the image segmentation results in the result output section. is entered. In the characteristic prediction unit, a regression model was constructed by SVM (Support Vector Machine), with the tag information as input and the tensile strength (MPa) as output. The steel No. shown in Table 1 is a number for managing tag information, and the "teaching data" and "test data" are teaching data used for learning the regression model, or for verifying the accuracy of the regression model. It shows the classification of the test data used for In the sense of showing the versatility of the constructed regression model, the results of tensile tests on both teacher data and test data are shown. The tensile strength was measured according to JIS Z 2241:2011 using a No. 5 test piece of JIS Z 2241:2011. Tensile test pieces were taken from the 1/4 part from the edge in the width direction of the sheet. Table 1 shows the strength prediction results together with the measured strength values, and an average absolute error of 12.0% is obtained for the test data prediction results. Although the prediction result of the training data is shown for comparison, the accuracy is 8.80%, and it can be confirmed that the model is functioning normally without the possibility of over-learning or the like. In the study shown in Table 1, the number of teacher data is 20 and the number of test data is 5, but it is expected that higher accuracy can be achieved by increasing the number of data. As for the regression model, increasing the number of data makes it possible to use a complicated model such as a CNN (Convolutional Neural Network) model for learning, and further improvement in accuracy can be expected.

According to the embodiment and example of the present invention as described above, identification of the microstructure of steel, which conventionally required visual judgment by an expert, can be performed by combining a machine learning algorithm and an image processing algorithm. Alternatively, by analyzing the local analysis data, it is possible to automate and save labor, secure quantification, and widen the identification range. In addition, by associating tag information including data acquisition methods (OM, SEM, EPMA or EBSD) with image data or local analysis data and performing segmentation using a learning model constructed according to the tag information, Segmentation is performed using an optimal learning model according to changes in data acquisition methods and imaging conditions, ensuring versatility and quantitativeness in estimating the microstructure characteristics of steel materials, and improving usability.

Tag information associated with image data or local analysis data may also include imaging conditions such as, for example, SEM source type. For example, if the imaging conditions differ between the teacher data of the learning model and the data to be segmented, the accuracy of tissue identification may decrease. Since the learning model used for segmentation is updated based on the tag information when the accuracy is lowered, it is possible to prevent the accuracy from being lowered.

Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is obvious that a person skilled in the art of the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. Naturally, it is understood that it belongs to the technical scope of the present invention.

100... Analysis device, 101... Data acquisition unit, 102... Segmentation unit, 103... Model construction unit, 104... Post-processing unit, 105... Result output unit, 106... Reliability calculation unit, 107... Model update unit, 108... Data Correction instruction unit 109 Data correction unit 110 Characteristic prediction unit 121 Map data 122 Model data 123 Result data.

Claims (11)

a data acquisition unit that acquires map data representing the microstructure of the steel;
and a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data. a reliability calculation unit that calculates the reliability of the classification result by the segmentation unit;
The analysis device according to claim 1, further comprising: a model updating unit that updates said learning model according to said reliability. 3. The analysis device according to claim 2, wherein said model updating unit performs additional learning or re-learning of said learning model using said map data extracted based on said tag information. The model update unit extracts tag information common or similar to tag information associated with map data to be segmented, and adds the learning model using the map data associated with the extracted tag information. 4. The analysis device according to claim 3, which performs learning or re-learning. The analysis device according to any one of claims 2 to 4, wherein the model update unit changes an algorithm or network structure of the learning model based on the tag information. a reliability calculation unit that calculates the reliability of the classification result by the segmentation unit;
The analysis device according to any one of claims 1 to 5, further comprising: a data correction instructing section that instructs correction of the map data according to the reliability. a result output unit for outputting a result of classification by the segmentation unit to a database;
When a new classification result is obtained by the segmentation unit, the steel material represented by the new classification result based on the degree of similarity between the new classification result and the classification result already stored in the database. 7. The analyzing apparatus according to any one of claims 1 to 6, further comprising a property prediction unit that predicts properties of the microstructure of. The map data includes at least one of image data and local analysis data of the microstructure of the steel material,
The analysis device according to any one of claims 1 to 7, wherein the tag information includes an acquisition method of the image data or the local analysis data. 9. The analysis apparatus according to claim 8, wherein said tag information includes at least one of steel grade classification, composition, heat treatment, etching conditions of said steel material, and photographing conditions of said image data or said local analysis data. obtaining map data representing the microstructure of the steel;
and classifying elements of the map data into segments using a learning model constructed according to tag information associated with the map data. a data acquisition unit that acquires map data representing the microstructure of the steel;
A program for causing a computer to function as an analysis device, comprising: a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data.

Images