JP7473765B2 - Automatic region identification method for non-metallic inclusions and inclusion discrimination system incorporating the method - Google Patents

Description

The present invention relates to a method and system for identifying nonmetallic inclusions in steel materials by automatically dividing image regions using AI machine learning for images of nonmetallic inclusions. In particular, the present invention relates to a method for identifying nonmetallic inclusions by dividing and identifying the regions for each type of inclusion using deep learning.

In recent years, AI (artificial intelligence), especially semantic segmentation using deep learning, has been used in various fields such as autonomous driving technology, medical image processing technology, and industrial inspection methods. Convolutional Neural Networks (CNN) have become widely used as a method for dividing image regions using deep learning. CNNs are an effective means for dividing image regions because they have a convolutional layer, a pooling layer, and a fully connected layer, but they are not suitable for dividing images into small regions and generally have low spatial resolution.

Recently, a fully convolutional neural network (FCN) has been proposed, which specializes in segmentation by making all fully connected layers convolutional layers. However, the problem of low spatial resolution remains unresolved even in FCN. To address this issue, the application of U-Net, which is characterized by increasing spatial resolution by reflecting image information at the time of convolution in the image at the time of deconvolution, has been progressing in recent years (Non-Patent Document 1).

However, in steel, tiny foreign bodies called nonmetallic inclusions are generated during the manufacturing process and cannot be completely removed, remaining in the material. When steel is used for high-strength parts such as bearing steel, it is known that repeated stress concentration on these nonmetallic inclusions during use can cause cracks, which can then become the starting point for fatigue failure.

Various methods have been proposed to evaluate nonmetallic inclusions in steel, including the microscopy method, which involves observing mirror-polished steel samples with an optical microscope, as specified in Japanese Industrial Standards JIS G 0555 and ASTM E45.

In addition, a method has been proposed for evaluating nonmetallic inclusions in steel, which involves observing the inclusions with an optical microscope and predicting the maximum inclusion diameter within a standard volume through extreme value statistical evaluation based on the size data of the inclusions observed within a standard inspection area (see Non-Patent Document 2).

In addition, in order to automatically distinguish between nonmetallic inclusions and factors that could be mistaken for nonmetallic inclusions, an inspection device has been proposed that calculates the features of the object from the captured image and distinguishes between nonmetallic inclusions present in metallic materials and things that are not nonmetallic inclusions but could be mistaken for nonmetallic inclusions (factors that could be mistaken for nonmetallic inclusions), such as dust particles or stains present on the test surface (see Patent Document 1).

Japanese Patent Application Laid-Open No. 10-62357

Olaf Ronneberger et al., "U-net: Convolutional Networks for Biomedical Image Segmentation," MICCAI, pp. 234-241 (2015) "Metal Fatigue: Effects of Micro Defects and Inclusions" Yokendou, by Takayoshi Murakami (1993)

Now, the range affected by stress concentration around nonmetallic inclusions is thought to correlate with the size of the nonmetallic inclusions. Therefore, when estimating the lifespan of a bearing, it is important to evaluate and understand the size of nonmetallic inclusions in steel. Furthermore, from the perspective of ensuring the reliability of the steel, it is important to understand the diameter of particularly large nonmetallic inclusions (maximum inclusion diameter).

Furthermore, based on the appearance of the inclusions (color tone and shape), the evaluator judges and classifies the type of inclusion as oxide, sulfide, or other inclusion. Of these inclusions, oxide-based inclusions are thought to have a particularly large effect on the service life, so it is important to be able to properly evaluate the cleanliness of oxide-based inclusions.

However, the method described in Patent Document 1 automatically distinguishes between nonmetallic inclusions and dust and stains that can lead to misidentification of nonmetallic inclusions, but does not further separate the nonmetallic inclusions themselves, so it cannot be said to be for distinguishing the types of nonmetallic inclusions, and does not address the identification of the types of inclusions by identifying regions for each of the oxides, sulfides, and other types of inclusions. Therefore, there is a demand for an efficient means for appropriately distinguishing the types of inclusions and measuring and evaluating the inclusion diameter for each type of inclusion.

Incidentally, the predicted inclusion diameter (√area _max ) calculated by the extreme value statistical evaluation is also an index of the quality of steel refining. In the extreme value statistical evaluation, an evaluator first observes a steel material that has been finished to a mirror finish using an optical microscope, identifies the largest inclusion within a specified reference observation area, and takes a picture of it. The same procedure is performed for a plurality of reference areas to obtain size data for a plurality of inclusions. Next, based on the obtained size data for the largest inclusion within the plurality of reference areas, the inclusion diameter is predicted by a method of predicting the inclusion diameter by applying the extreme value statistical method.

However, the identification of the regions of each type of inclusion using this method presupposes the skill of the evaluator, both in identifying the type of inclusion and grasping the size. This means that it is highly dependent on the skill of the evaluator, so even if the method is the same, the accuracy depends largely on the knowledge and experience of the evaluator. Although experience is required to identify the type of inclusion, grasping the size and shape of the inclusions in extreme value statistical evaluation is not easy. This is because the shape of inclusions in steel is not always simple circular, but is often shaped like elongated islands. Furthermore, when the inclusions are intermittently and continuously scattered like islands, they must be interpreted as a single inclusion, so they require skill. Therefore, the identification of the shape and size of such inclusions is prone to variation depending on the evaluator, and as a result, even if the evaluation is performed using the same method, the results obtained may vary. In addition, it takes time to train skilled personnel, and it is not easy to train and secure human resources.

The problem that the present invention aims to solve is to provide a means for accurately estimating the range of each type of inclusion using deep learning and automatically dividing and discriminating from an image in order to appropriately and stably measure the maximum size of each type of inclusion, which is necessary for processing the extreme value statistics method based on an image containing nonmetallic inclusions in steel, in order to solve these problems related to the evaluation of the size of nonmetallic inclusions.

In order to solve the above-mentioned problems, the applicants have obtained an inclusion discrimination method that automatically divides and identifies an image of a nonmetallic inclusion into regions of various inclusion types by using a learning model generated by a computer through machine learning using training data based on a group of learning data sets that include multiple data pairs of optical microscope images of steel material containing nonmetallic inclusions to be discriminated, which are raw data, and colored images, which are correct answer data in which the inclusions to be discriminated are colored according to their various inclusion types. When an optical microscope image of a steel material in which inclusion regions are unidentified (unknown) is queried, the feature amounts of the input image are extracted, the inclusion types are discriminated, and the inclusions are colored according to their inclusion types.

That is, the method includes the steps of, in order, having a computer learn by machine learning a set of learning data sets, which are teacher data, and which include a plurality of data pairs consisting of optical microscope image data (raw data) including nonmetallic inclusions to be identified in the steel material to be evaluated and colored image data (ground truth data) in which the image regions of the inclusions to be identified are colored according to the type of inclusion (model learning step), a step of extracting the feature values of the input image based on machine learning when an unknown optical microscope image, i.e., an inclusion region is not yet identified (feature extraction step), a step of identifying the inclusion type based on machine learning (inclusion identification step), and a step of outputting the inference result as a identified image colored according to the type of inclusion (identified image output step), and is characterized in that the image regions of nonmetallic inclusions are automatically identified by AI and divided and identified into various inclusion regions.

In the machine learning process for generating trained models, high-resolution grayscale images are used as raw data from optical microscope images, and a large number of colored images, which are colored by type of inclusion using image editing software based on the results of chemical composition analysis using characteristic X-ray analysis, are prepared as paired training data and used to generate trained models.

Furthermore, the accurate data set obtained in this way cannot be obtained in large quantities or easily because the creation process is complicated and requires a large workload. Therefore, the amount of training data can be increased by performing data expansion such as performing geometric operations on the training data image, such as inverting, cropping, enlarging, and rotating, or performing operations such as brightness, contrast, and adding noise.

As a machine learning method for generating a trained model, U-Net and its derived methods can be used. Then, pseudocolor is introduced to grayscale images, which are raw data for training, to input data in three-channel RGB color for high-precision training in U-Net. As a loss function, it is advisable to adopt an adjusted weighted cross-entropy loss function, which allows weighting to be set for inclusion systems, for which accuracy in area identification is important. By using this as a loss function, the inference result and the correct answer are input to the loss function during model training, and the parameters are adjusted so that the error between the two is reduced, thereby making it possible to obtain an inference result closer to the correct answer.

That is, the first means for solving the problem of the present invention is a method for discriminating inclusion types, which includes, in order, a step of generating a trained model by machine learning using training data obtained based on a group of learning data sets including a plurality of data pairs each consisting of optical microscope image data of a steel material containing a nonmetallic inclusion to be discriminated and colored image data in which the image regions of the nonmetallic inclusions to be discriminated corresponding to the optical microscope image data are colored in different colors for each inclusion type by using an electronic computer, and a step of extracting feature amounts related to an optical microscope image (query image) of the steel material input based on the trained model by the electronic computer, a step of discriminating the type of inclusion based on the trained model, and a step of acquiring colored image data in which the image regions are colored in different colors for each inclusion type based on the discrimination result.

The second means is the method for identifying an inclusion system described in the first means, characterized in that the teacher data includes data pairs generated by data expansion based on the data pairs of the learning data set.

The third means is a method for determining the diameter of an inclusion as described in the first or second means, characterized in that the optical microscope image data is a grayscale image, and the optical microscope image is converted into a pseudo-color converted image of a three-channel RGB color image, and then the pseudo-color converted image is subjected to machine learning together with the colored image data of the data pair to generate a trained model.

The fourth means is a method for distinguishing inclusion types described in any one of the first to third means, characterized in that the colored image data in which different colors are used for the data pairs of the training data and in which different colors are used for each inclusion type based on an optical microscope image of the inclusion that has been subjected to color inversion processing is used to color image data in which different colors are used for each inclusion type.

The fifth means is a method for discriminating inclusion systems described in any one of the first to fourth means, characterized in that the trained model is generated by machine learning using a U-Net full-layer convolutional neural network.

The sixth means is a method for discriminating an inclusion system according to any one of the first to fifth means, characterized in that the trained model is generated by machine learning using a weighted cross-entropy error as a loss function, adjusting the weight of the type of inclusion to be discriminated so that it is large.

The seventh means is a method for distinguishing inclusion types described in any one of the first to sixth means, characterized in that it includes a step of measuring the major and minor diameters of the inclusion types to be evaluated and calculating the size of the inclusions based on a colored image in which the inclusion types are colored differently based on the discrimination results.

The eighth means is a method for discriminating inclusions according to any one of the first to seventh means, characterized in that it includes a step of measuring the major and minor axes of the inclusion system to be evaluated based on a colored image in which each inclusion system is colored differently based on the discrimination result, calculating the size of the inclusion, selecting the largest inclusion in the steel material in the observed area, and outputting the size √area of the largest inclusion.

The ninth means is a method for discriminating inclusion types according to the eighth means, characterized in that it includes a step of calculating the maximum inclusion size √area for each of a plurality of test pieces having a predetermined observation area, and calculating the predicted inclusion diameter (√area _max ) in a predetermined predicted area of the evaluation steel material using extreme value statistics based on the obtained data on the sizes of the plurality of maximum inclusions.

The tenth means is an inclusion discrimination system equipped with the inclusion discrimination method described in any one of the first to ninth means.

The other means is a method for distinguishing inclusion types described in any one of the first to sixth means, characterized in that it includes a step of measuring the major axis of the inclusion type to be evaluated and calculating the size of the inclusion based on a colored image in which the inclusion type is colored according to the discrimination result.

The method according to the present invention makes it possible to appropriately distinguish the inclusion type area from within an optical microscope image that contains unknown inclusions, etc., that are the subject of a query, based on the generated trained model, and to display the image area of the inclusions in a different color for each type of inclusion.

1 is a flow chart showing an outline of the steps of carrying out the present invention. 1 is an image of an inclusion taken with an optical microscope. A diagram showing the structure of U-Net used in the learning model. 4 is an enlarged image showing the procedure for creating images with different colors for each type of inclusion. (a) is a color image taken with an optical microscope through a green filter, (b) is an enhanced image in which the contours of the inclusions are emphasized, and (c) is an image with different colors for each type of inclusion (note that the original image of FIG. 4 is a color image). This is an enlarged view of an inclusion image captured by an optical microscope and a colored image after inference in which the inclusion type is colored differently (shown in grayscale, but the original is a color image). This figure shows the difference in oxide area determination and the difference in the measurement results of the oxide size (√area: square root of the product of the minor axis and major axis of the inclusion) based on that. (a) is an example of a conventional method in which a person determined the area based on a monochrome image of an optical microscope photograph, and the area was determined to include sulfides. (b) is an example in which the size of the oxide was correctly determined. This is a diagram in which multiple oxide diameters plotted based on extreme value statistics are organized based on the correct data. The plots indicated with a black circle symbol are predictions based on the AI of the method of the present invention, and the plots indicated with a circle symbol are examples of predictions based on human measurements using the conventional method.

The AI that automatically divides the image region of inclusions according to the present invention, that is, the process of automatically dividing and identifying the image region for each inclusion type by machine learning and outputting it, will be described below with reference to the flowchart in Figure 1. Also, in the following examples, the explanation focuses on oxides as a representative main discrimination target, but a similar approach can be used for sulfides as well as oxides, and the shape and size of each inclusion type can be appropriately captured based on the generated learning model for optical microscope images that contain unknown inclusions, etc.

(Step A: Collecting test pieces)
In preparing the test pieces for use in extreme value statistical evaluation, the steel material to be evaluated is subjected to an appropriate heat treatment as necessary so that cutting and cutting can be easily performed, and then the test pieces are cut out from the steel material so that a predetermined observation area can be obtained. In addition to adjusting the polishing conditions, if inclusions are likely to fall off during finish polishing in the subsequent process, or if polishing scratches are likely to remain, the hardness of the steel material may be increased to improve the situation, so the hardness may be adjusted by heat treatment as necessary. Next, the test pieces are left as they are, or if necessary, the test pieces are embedded in resin, and then the observation surface of the sample is subjected to finish polishing to prepare test pieces for optical microscope observation. A plurality of such test pieces are prepared.

(Step A: Specific Example)
In the examples, JIS SUJ2 steel was used as the steel material to be evaluated, and a plurality of test pieces having a size of 10 mm x 10 mm x 8 mm were taken from the vicinity of the middle circumference of a round bar made of SUJ2 steel that had been hot-forged to φ65 mm in order to perform extreme value statistical evaluation. The area of the observation surface at this time was 100 ^mm2 . In this example, the inclusion evaluation was performed from the vicinity of the middle circumference of the steel material, so the test pieces were taken from that location, but if it is desired to perform inclusion evaluation near the surface or center of the steel material, it is sufficient to take the test pieces from that location. The test pieces were taken so that the surface parallel to the rolling direction was the observation surface of the sample. Then, the taken test pieces were embedded in resin, and then polished using an automatic wet polishing device or a manual wet polishing machine, and finally buffed using diamond abrasive grains to process into a mirror state. Note that, following the buffing, finishing polishing using a colloidal silica solution may be performed.

(Step B: Imaging of non-metallic inclusions)
For the inclusions to be evaluated in each test piece obtained in step A, the inclusion with the largest inclusion size √area (the square root of the product of the short diameter and the long diameter of the inclusion) is identified. If the largest inclusion is a composite inclusion (an inclusion composed of multiple inclusions with different chemical compositions), it may not be the largest inclusion to be evaluated as a result of the inclusion area identification described below, so it is advisable to identify the next largest inclusion and, if necessary, the next largest inclusion as an evaluation target. Next, for the inclusions (single composition inclusions, composite inclusions, and inclusions that may be treated as a single entity such as inclusions with a shape that is intermittently and continuously scattered like islands), the observation magnification of the optical microscope is set to a high magnification so that the fine shape and color difference can be recognized, and then an image of the inclusion to be evaluated is taken. At this time, it is preferable to select an image size and file saving format with sufficiently high resolution.

(Process B: Specific Example)
FIG. 2 shows an image of an inclusion captured by an optical microscope. The image is in grayscale, and the observation magnification is unified at 400 times to be used as training data in the subsequent process C. In order to ensure that the image resolution is sufficiently high, the image size is set to 4080 x 3072 pixels, and the file storage format is selected to be the BMP format. The magnification of the image may be adjusted so that the entire inclusion fits within the image depending on the size of the inclusion. In addition, the image size and storage format of the captured image may be selected to be lower in resolution as long as they do not cause a decrease in the accuracy of machine learning.

(Process C: Creating teacher data)
As training data for creating a learning model, images for a data set (learning data set) are prepared, which are a set of data pairs each consisting of an inclusion image (raw data) obtained in step B and a colored image (correct answer data) in which the raw data is colored according to the type of inclusion based on the results of the chemical composition analysis of the inclusions using image editing software. In addition, training data may be obtained by implementing data expansion to efficiently perform highly accurate learning with a small data set.

(Step C: Specific Example)
The colored images in which the inclusions are colored according to their type and used as the correct answer data in the learning data set were created by determining the inclusion types from the results of composition analysis based on characteristic X-rays obtained by an energy dispersive X-ray spectrometer (EDS) or an electron probe microanalyzer (EPMA), and then coloring the inclusions by type using a commercially available general image editing software, using a photographed image of the raw data as a base image. As shown in Figure 4, inclusion types such as oxides, sulfides, and contaminants are identified and colored differently.

In addition, the different colors referred to in this invention are sufficient if they can be distinguished and recognized by a computer, and therefore the different colors do not necessarily need to have clear differences in color to the human eye. In other words, specifically, the different colors may be used to paint in different hues, such as red and green, for each type of inclusion, or may be differences in shades such as light gray and dark gray, or differences in saturation for red-based colors. Also included are images in which a translucent color image is overlaid on top of the original image for each type of inclusion to create differences, and image data in which a layer is used on the original image to compositely display colors for each type of inclusion are also included in the different colors referred to here.

When creating training data, for example, the accuracy of inclusion type discrimination in the training data itself can be improved by taking the following measures, making it easier to obtain better machine learning results.

First, in addition to the grayscale image used for the data pair of the optical microscope photograph, an image of the inclusions is also taken through a single-color transparent color filter. As the color filter, for example, a green filter can be used, but it is not limited to this. Next, in order to further emphasize the contrast of the contours of each type of inclusion, a commercially available image editing software (such as Adobe Photoshop (registered trademark)) is used to perform image processing to emphasize the inclusions, such as manipulating the color tone and gradation by performing a solarization process (color inversion process) using the color curve correction function implemented in the software. In addition, a conversion process is appropriately performed to maximize the brightness of the midtones while minimizing the contrast of the midtones.

Based on the color image obtained by the image processing as described above, the inclusion area is appropriately identified, and then the texture of the inclusion to be evaluated is observed to determine the color to be painted. A scanning electron microscope (SEM) was used for the texture observation. A backscattered electron image of the inclusion was obtained at an appropriate magnification with the SEM, and the inclusion type was confirmed with EDS. The reason why the backscattered electron image is used instead of the secondary electron image used in normal SEM observation is that it is highly useful, such as obtaining contrast according to the chemical composition and being less susceptible to the effects of charge-up during SEM observation in inclusions with high electrical insulation. By comparing this image and understanding the inclusion type and representative composition with EDS, the area of each inclusion type in the processed color image was identified, and color-coded by inclusion type such as oxide-based and sulfide-based inclusions, and colored image data was obtained. The above grayscale image of the inclusion taken by the optical microscope and the colored image data created by color-coding by inclusion type were made into data pairs. A large number of such data pairs were created to obtain a data set.

The dataset obtained through these procedures can be said to be more advantageous in improving the accuracy of deep learning, the more data there is in the dataset. However, since the task of determining correct data is a heavy workload and the task itself requires skill, it cannot be said that this data can be obtained in large quantities and easily by simply increasing manpower. Therefore, in addition to using only these datasets as training data, the training data can be further enriched by expanding the data based on the dataset as follows:

(About data expansion)
By performing geometric operations such as inversion, cropping, enlargement, and rotation on the prepared data set, and also performing color space operations such as brightness, contrast, and noise addition, the number of data used for learning can be expanded, making it possible to perform more accurate learning even with a small data set. Therefore, data expansion was performed based on the created data set. For data expansion, a large data set may be prepared in advance for processing determined by offline expansion, but in order to reduce the amount of storage used by the electronic computer, in the present invention, data expansion is performed on one mini-batch by on-the-fly expansion (online expansion) using geometric operations and color space operations randomly selected just before, and the machine learning of the step D shown below is performed sequentially. If multiple epochs (the number of repeated learning of one training data) are used for learning, different images are randomly used from the same mini-batch, so it can be performed without putting pressure on the storage capacity.

Specifically, 864 optical microscope images were prepared. First, these images were randomly assigned: 724 for training, 70 for validation, and 70 for evaluation. All training data was learned in one epoch, and the data expansion operation for each epoch was selected randomly. Finally, after 100 epochs of learning, the training data was expanded to 72,400 images. Furthermore, some training data may be created by semi-supervised learning and used for machine learning.

(Step D: Model learning and evaluation)
In deep learning, a machine learning technique using a computer, for example, a pre-trained model (pre-trained model) of ImageNet incorporating a convolutional neural network (CNN) is used for transfer learning in the convolutional layer of U-Net to learn a data set of non-metallic inclusions. After these learning steps, accuracy evaluation is performed using validation data to verify the validity of multiple models. Based on the accuracy evaluation using the validation data, the best model is determined, and a final evaluation is performed using the evaluation data.

Here, we briefly explain transfer learning. Transfer learning is a technique that applies knowledge from one domain to the learning of another domain, and is one of the machine learning techniques that makes it possible to reduce the time required to acquire large amounts of data and to learn by repurposing trained models such as those described above. Therefore, the use of transfer learning has the advantage that it allows target images to be identified with high accuracy even when there is little training data.

(Step D: Specific Example)
FIG. 3 is a diagram showing the structure of the U-Net neural network model used in deep learning. U-Net is a neural network model characterized by enhancing spatial resolution by reflecting information at the time of convolution in the image at the time of deconvolution. In this model, the convolution layer (encoder) convolves the input image multiple times while extracting features contained in the image, while the deconvolution layer (decoder) performs the reverse process to that at the time of convolution to output image data of the same size (same resolution) as the input image. Here, in the feature extraction in the encoder, a pre-trained model using ImageNet, which is a large-scale teacher-labeled image dataset, can be transferred and used as is, so in the embodiment, the convolution neural network VGG19 with 19 convolution layers, which has the best results in the model accuracy evaluation index described later, was used for transfer learning.

In the present invention, in order to compare the inferred output image with the input image at the same size, the convolutional layer on the encoder side has a filter size of 3×3, a stride of 1, a padding of 1, and max (maximum value) pooling is performed using a 2×2 filter, while the convolutional layer on the decoder side has a filter size of 3×3, a stride of 1, a padding of 1, and upsampling of bilinear interpolation is performed using a 2×2 filter as the inverse of pooling. Note that average pooling may be selected as the pooling method, and Nearest Neighbor Interpolation may be selected as the upsampling method. In addition, ReLU, which is often used in deep learning and is compatible with image data, is adopted as the activation function that determines the output of the neural network. Note that LeakyReLU, PReLU, or ELU may be selected as the activation function instead of ReLU.

To implement the convolutional neural network (CNN), Keras was used with the deep learning framework TensorFlow (tensorflow-gpu 2.4.0) as the backend, and the computer operating environment was Windows 10 Pro OS, Corei 7-9700K CPU, and RTX 3090 GPU. Of course, these specifications are just examples and may be changed as appropriate when implementing the present invention.

Regarding learning, the image size of the training data of inclusions obtained in step C was reduced to 2040 x 1536 pixels to speed up the learning speed. Furthermore, since most of the image taken with an optical microscope is the background and the target inclusion is located in the center of the image area, the input image was divided into 512 x 512 pixels in size to prevent excessive learning, and images without inclusions were removed. Furthermore, images were normalized to stabilize learning and improve generalization performance. In addition, pseudocolor was introduced as a preprocessing before deep learning. By introducing pseudocolor, it is possible to emphasize the type of inclusion that is important, improving the identification accuracy and enabling efficient learning. For the image processing required for pseudocolor, OpenCV, a free library for computer vision (a research field for automating tasks that the human visual system can perform with machines), was used.

After inference, the images divided into 512 x 512 pixel pieces are soft-voted (average of predicted probability of overlapping parts) and merged to an image size of 2040 x 1536 pixels. At this time, the size of the divided images is equal to the U-Net's input image size of 512 x 512 pixels. It is also equal to the U-Net's output image size of 512 x 512 pixels after inference. The output images are shifted by 256 pixels up, down, left, and right, and then merged with overlap to obtain a merged image of 2560 x 2048 pixels. The 256 pixel margin on one side that was generated during image merger was deleted, and the images were merged to a size of 2040 x 1536 pixels.

Next, we will explain the reason for introducing pseudocolor in detail. In semantic segmentation using deep learning such as U-Net, it is possible to transfer learn a pre-trained model using ImageNet, which is a large-scale database of teacher-labeled images of color photographs, in feature extraction in the encoder. Therefore, in order to perform deep learning of optical microscope images such as this one using semantic segmentation such as U-Net, it is desirable to preprocess the optical microscope images so that they match the shape and data structure of the color images in the image database to be used for learning.
Therefore, in the case of one-channel grayscale images, which are often used as optical microscope images, a common preprocessing step is to convert the one-channel grayscale image into a three-channel RGB color image by superimposing the three channels.
However, the converted three-channel image does not contain the original RGB color information, so most of the filters in the convolution layer close to the input layer in U-Net and other systems overlap, making it impossible to make effective use of the three-channel information and learning.

Therefore, in the present invention, pseudocolor conversion is used for preprocessing, which converts a one-channel grayscale image into a three-channel RGB color image by applying an appropriate conversion map. A desirable conversion map is one that easily emphasizes oxides and other elements that are important for identification, such as a color map called Jet that can convert grayscale into rainbow colors.

The introduction of pseudocolor improved the accuracy of identifying various inclusion types compared to the results of deep learning using only grayscale images taken with an optical microscope.
In addition, just as in the case of creating a learning model by performing preprocessing using pseudo-color conversion, when identifying the type of inclusion in an unknown optical microscope image, the grayscale image can be pseudo-colorized and then features can be extracted, allowing the inclusion type in the query image to be identified by a computer using the learned model.

Furthermore, when performing deep learning such as U-Net, a loss function is adopted that evaluates the difference between the output inference result and the correct answer data in the training data and changes the weighting of the neural network accordingly. There are various options for this loss function, but since the purpose of the present invention is to divide and identify the image region for each type of inclusion system, an adjusted weighted cross entropy loss function is adopted. This solves the problem (called class imbalance) of an optical microscope image of non-metallic inclusions sparsely present in mirror-polished steel, where the steel parts other than the specific inclusions (which are imaged as white in the optical microscope image, and can be considered as the background, and are not involved in identifying the image region of the inclusions) occupy most of the image by adjusting the weight of the background. The weights were calculated by calculating the total number of pixels in each image region, such as the background and inclusions, and the reciprocal of this was used as a reference value, and were adjusted based on the reference value.

As an accuracy evaluation index for deep learning, an index called mIoU (mean Intersection over Union) was used. This is an index showing how close the inference result is to the correct answer, and evaluates the ratio of correctly determined areas to the area where the inferred inclusion system area and the inclusion system area in the correct answer image are superimposed, so it is possible to evaluate the degree of accuracy of the inference result without being affected by the ratio of the size of the inclusions to the entire image (which is often a small ratio).
In addition, when using Pixel Accuracy, which is also relatively often used as an accuracy evaluation index for segmentation, for accuracy evaluation, in the case of a photographed image of an inclusion with class imbalance, in which the background occupies the majority of the pixels in the entire image as described above, the influence of prediction error in a small class (here, referring to inclusions) is relatively small compared to a large class (here, referring to the background), so that as a result, it is easy to consider that Pixel Accuracy is high, that is, the accuracy of inference is high, and it is not suitable for grasping the change in accuracy due to model tuning. Therefore, in the verification, we mainly focused on the mIoU index.

Table 1 shows the evaluation results of each model in terms of Pixel Accuracy and mIoU on the validation data. In Table 1, TL indicates that transfer learning was performed, and FT indicates that fine-tuning was performed by making partial adjustments to the existing model. It can be seen that even when there is almost no change in Pixel Accuracy, there is a change in mioU. The mIoU of the learning result of the original U-Net (the 10th model from the top in Table 1), shown in bold and underlined, is 0.682, while the mIoU of the model in which VGG19 was transferred to U-Net and learned (the model in the first row in Table 1) is 0.745, which is a significant improvement of more than 6% in mIoU. Therefore, further tuning was performed based on this model.

By using the above-mentioned adjusted weighted cross entropy loss function for a model in which VGG19 was transferred to U-Net and the weights for each type of inclusion were tuned, the mIoU was further increased to approximately 0.81.

(Step E: Discrimination of various inclusion types for input images in which the type of inclusion is unidentified (unknown))
An inclusion image (raw data) in which the inclusion region is unidentified (unknown) photographed by an optical microscope is input to the learning model obtained by the process D, and a colored image in which various inclusion types are colored differently is output as an inference result. Note that, since the coloring here means that it is sufficient for an electronic computer to distinguish and recognize the different colors, it is not necessarily required that the difference in color tone is clearly visible to the human eye.

(Step E: Specific Example)
An image captured by an optical microscope and an enlarged view of the colored image after inference are shown in Figure 5. Here, a one-channel grayscale image captured by an optical microscope, in which the inclusion region is unidentified (unknown) and not used in deep learning, is converted into a three-channel RGB color image using pseudo-color conversion as a preprocessing step, as in the case of creating a learning model, and then input into the model of the present invention, making it possible to identify the oxides, sulfides, and contaminants contained in the image and automatically color them differently.
In the embodiment, oxides are colored red, sulfides are colored blue, and contaminants are colored green. Note that the image size is 4080 x 3072 pixels, and the file is saved in BMP format, but the image size and file save format may be changed depending on the situation.

The method according to the present invention in steps A to E makes it possible to accurately identify image regions of inclusions according to various inclusion types from optical microscope images in which the types of inclusions are unidentified (unknown) and to automatically color them differently, thereby making it possible to evaluate the size of inclusions using the extreme value statistics method by AI without human intervention.
As shown in the procedure for creating training data in Figure 4, in the past, color-coded images were created by combining equipment and manually distinguishing between them. However, by using the trained model generated by the present invention, it is possible to automatically identify the type of inclusion and obtain image data in which the target image area is appropriately colored, and a colored image like that shown in Figure 4(c) can be automatically obtained.

The 70 optical microscope images were examined as follows to see whether they could not only capture inclusions but also properly distinguish the types of inclusions within the inclusions.
First, 70 images captured by an optical microscope containing inclusions whose inclusion types were previously identified were prepared and were classified using the trained model of the present invention.
The breakdown of the foreign matter in the 70 images was as follows: the foreign matter (inclusion) with the largest area was oxide: 33 images, sulfide: 19 images, and foreign matter other than inclusion: 18 images.
Results: When these images were discriminated using the trained model of the present invention, all of the inclusion types in the inclusion region were properly identified, and the inclusion types with the largest area were correctly determined to be oxides: 33 images, sulfides: 19 images, and foreign matter other than inclusions: 18 images. In other words, when the trained model was used, the inclusion types in the region with the largest inclusions were properly identified.
In this way, it was confirmed that by using the deep learning machine learning model of the present invention, it is possible to properly identify the type of inclusions among inclusions in an optical microscope image without the need for work by an expert.

(Process F: Implementation of extreme value statistics)
For multiple test pieces taken from the steel material to be evaluated, the maximum size of the inclusion in each test piece is calculated as √area (the square root of the product of the minor axis and major axis of the inclusion) based on the color-coded images of the inclusions obtained through the above process (colored images in which the areas of each inclusion type are identified).
In the following examples, the area is used to calculate the size of the inclusions. However, the size of the inclusions may be evaluated by discriminating the inclusion system by using only the major axis (maximum length) of the inclusions instead of the area.

Based on the obtained data on the size of the largest inclusion, an evaluation is performed using extreme value statistics to predict the maximum inclusion diameter √area _max that can be contained in any predicted volume of the steel material.

If the inclusion type cannot be correctly identified, oxides and sulfides will not be distinguished as in Figure 6(a), and the determination of √area will be inappropriate. When determining the √area of oxides, as in Figure 6(b), it is necessary to exclude sulfides before identifying them. However, it is not easy for a person to distinguish between them from a grayscale image such as that in Figure 6. By using an image in which the inclusion type has been automatically and appropriately identified using the procedure of the present invention, the √area can be measured appropriately.

(Process F: Specific Example)
Based on the colored images colored for each inclusion type in step E, √area was measured and the extreme value statistics method was performed.

Here, the √area measurement of the colored images colored for each inclusion type may be performed by incorporating rule-based AI using OpenCV or the like into the learning model.

The extreme value statistics method used to predict the inclusion diameter will be described. In this case, the extreme value statistics method involves taking a plurality of test pieces (j=1, n) from a certain population, photographing the largest inclusion present within a predetermined observation range of each test piece with an optical microscope, plotting the size of the largest inclusion in each test piece in ascending order of size on an extreme value statistics graph, and predicting the maximum inclusion diameter (√area _max ) in any area to be predicted based on an approximation line of the plots. The obtained √area _max can be used for product quality inspection, as a comparative index for the inclusion cleanliness of steels to be evaluated, or as an index for improving cleanliness by improving the steel refining method.

The value of the cumulative distribution function F (unit: %) shown on the vertical axis in the extreme value statistics graph can be obtained from the formula F _j = j/(n+1) × 100. Here, n indicates the total number of test pieces for which √area was evaluated, and j indicates the jth test piece when arranged in ascending order of √area. However, in this case, it is necessary to use a probability paper to determine the plot position on the vertical axis. Therefore, instead of F, the vertical axis may be represented by using the value of y _j obtained from the formula y _j = -ln [-ln {j/(n+1)}] as the standardized variable y.
The plotting method is to plot in ascending order of the maximum inclusion size √area in the observation field, with the horizontal axis value being √area and the vertical axis value being the normalized variable y _j corresponding to the jth inclusion.
Next, in order to predict the maximum inclusion diameter (√area _max ) that can be present in an arbitrary area, the cumulative distribution function F and normalized variable y in the area to be predicted are obtained from the recursion period T. In this case, T is obtained from the predicted area S and the observation area S ₀ of each test piece by the formula T=(S+S ₀ )/S ₀ , and the cumulative distribution function F can be obtained from the formula F=(T-1)/T×100 using this T. Also, the vertical axis may be represented by the value of y obtained from the formula y=-ln[-ln{(T-1)/T}] as the normalized variable y. Furthermore, when S>>S ₀ , y=lnT and T=S/S ₀ hold, which makes it possible to simplify the calculation.

Thus, as an example, for 30 test pieces, images of the largest inclusions present within a predetermined observation range were captured using an optical microscope, and the trained model generated through steps A to D was used to identify the inclusion type in each image to obtain identified images with different colors. After that, the maximum inclusion diameter √area _max in an arbitrary area was determined by performing extreme value statistical processing using each measurement result of the maximum inclusion diameter √area of the oxide inclusions measured based on the obtained images.

Furthermore, Figure 7 shows a diagram in which the extreme value statistical results obtained based on the AI method of the present invention and the extreme value statistical results obtained based on manual judgment by a person (evaluator) using conventional normal procedures are organized using the correct data as the standard. The results show that in advanced judgment involving oxide integration judgment, AI has led to results that are closer to the correct data than the previous manual judgment by a person, and it has been confirmed that an automatic judgment system can be used to make judgments that are equal to or better than those made by an experienced person.

According to the method of the present invention, √area can be calculated for each automatically determined inclusion type, and these values can be used to further calculate √area _max by the extreme value statistics method. Since √area for each inclusion type can be grasped, prediction accuracy can be improved and there is no variation due to the level of skill of the operator, as occurs when the inclusion type is determined manually.

The √area for each type of inclusion and the √area _max calculated based on it may be output as data such as a CSV file.

As described above, the method according to the present invention in steps A to E uses the generated trained model to accurately identify image regions of inclusions by type from optical microscope images in which the type of inclusion is unidentified (unknown), and captures the regions so that they can be automatically colored. These steps are then performed on multiple test pieces, and by combining them with step F to perform extreme value statistical evaluation, a prediction result for the maximum inclusion diameter can be obtained.

According to the present invention, such a series of procedures can be carried out extremely efficiently and accurately using AI, so that the inclusion type can be identified by importing and referencing optical microscope images from the steel material to be observed into a trained electronic computer without the need for manual judgment by a skilled person as in the past, and identification results can be obtained quickly with almost the same accuracy as that of a skilled person. Then, if the maximum inclusion diameter √area for a desired inclusion type is obtained for a plurality of test pieces, the maximum inclusion diameter √area _max can be predicted using extreme value statistics based on these values, making it possible to make an extremely efficient and accurate prediction.

Claims (10)

generating a trained model by machine learning a computer using training data obtained based on a group of learning data sets including a plurality of data pairs each including optical microscope image data of a steel material including a non-metallic inclusion to be discriminated and colored image data in which an image region of the non-metallic inclusion to be discriminated corresponding to the optical microscope image data is colored in a different color for each type of inclusion;
A step of extracting feature quantities related to the input and referenced optical microscope image of the steel material by a computer based on the trained model;
A step of identifying the type of inclusion based on the trained model;
acquiring colored image data that is colored for each inclusion type based on the discrimination result;
A method for identifying inclusion types, in which an image including nonmetallic inclusions is automatically divided and identified by machine learning into image regions for each inclusion type by a computer.
The optical microscope image data used to generate the trained model is a grayscale image,
The optical microscope image is converted into a three-channel RGB color image to create a pseudo-color converted image, and then the pseudo-color converted image is subjected to machine learning together with the colored image data of the data pair to generate a trained model. This is a method for discriminating inclusion types in an image containing non-metallic inclusions, in which an electronic computer uses machine learning to automatically divide and identify image regions for each inclusion type. generating a trained model by machine learning a computer using training data obtained based on a group of learning data sets including a plurality of data pairs each including optical microscope image data of a steel material including a non-metallic inclusion to be discriminated and colored image data in which an image region of the non-metallic inclusion to be discriminated corresponding to the optical microscope image data is colored in a different color for each type of inclusion;
A step of extracting feature quantities related to the input and referenced optical microscope image of the steel material by a computer based on the trained model;
A step of identifying the type of inclusion based on the trained model;
acquiring colored image data that is colored for each inclusion type based on the discrimination result;
A method for identifying inclusion types, in which an image including nonmetallic inclusions is automatically divided and identified by machine learning into image regions for each inclusion type by a computer.
A method for discriminating inclusion types in which a computer automatically divides and identifies image regions for each inclusion type using machine learning for an image containing non-metallic inclusions, characterized in that the colored image data in which each inclusion type is painted in a different color and used in the data pairs of training data used to generate the trained model is colored image data in which each inclusion type is painted in a different color based on an image that has been color-reversed. 3. The method for discriminating inclusion systems according to claim 1 or 2 , wherein the teacher data includes data pairs generated by data expansion based on the data pairs of the learning data set. The method for determining an inclusion system according to claim 1, characterized in that the colored image data in which different colors are used for the data pairs of the training data and in which different colors are used for each inclusion type are colored image data in which different colors are used for each inclusion type based on an image that has been subjected to color inversion processing. The method for discriminating inclusion systems according to claim 1 or 2 , characterized in that the trained model is generated by machine learning using a U-Net full-layer convolutional neural network. The method for discriminating inclusion systems according to claim 1 or 2, characterized in that the trained model is generated by machine learning using a weighted cross-entropy error as a loss function, adjusting the weight of the type of inclusion to be discriminated so that it is large . 3. The method for identifying inclusion types according to claim 1 or 2, further comprising a step of measuring the major axis and minor axis of the inclusion type to be evaluated based on a colored image which is colored differently for each inclusion type based on the identification result, and calculating the size of the inclusion. 3. The method for discriminating inclusion types according to claim 1 or 2, characterized in comprising a step of measuring the major axis and minor axis of the inclusion type to be evaluated based on a colored image which is colored according to the discrimination result, calculating the size of the inclusion, selecting the largest inclusion in the steel material within the observed area, and outputting the size √area of the largest inclusion . 9. The method for discriminating inclusion types according to claim 8, further comprising the steps of: calculating the maximum inclusion size √area for each of a plurality of test pieces having a predetermined observation area; and calculating a predicted inclusion diameter (√area _max ) in a predetermined predicted area of the evaluation steel material using extreme value statistics based on the obtained data on the sizes of the plurality of maximum inclusions. An inclusion discrimination system comprising the inclusion discrimination method according to any one of claims 1 to 4.

Images