JP2023163983A - Image processing device, image processing method, image processing program, image classification device, and learned model generated by image classification device - Google Patents

Abstract

To provide an image processing device capable of creating data for achieving stable determination accuracy even when an amount of created data is small and used for machine learning, and an image processing method and the like.SOLUTION: An image processing device 10 comprises first arithmetic means 120 and second arithmetic means 130. The first arithmetic means numerically converts a photographic image O to a plurality of pieces of color information per pixel, and creates data by linking each of pieces of the numerically converted color information. The second arithmetic means creates data used for machine learning by subjecting the data created by the first arithmetic means 120 to fast Fourier transformation.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, an image processing method, and an image processing program that create data used for machine learning based on information acquired from captured images. The present invention also relates to an image classification device that performs machine learning using data created by these image processing devices, and a trained model generated by this image classification device.

Conventionally, methods are known for detecting abnormalities from images taken with a camera or the like. For example, by using deep learning to learn images that show normal parts without scratches and images that show abnormal parts with scratches as training data, the parts shown in a certain image are normal (no scratches). There is a known method for generating a trained model that determines whether the object is defective or abnormal (with scratches).

Detection methods (determination methods) based on captured images using such machine learning are required to have improved determination accuracy, be lightweight, and have real-time performance. In particular, achieving high judgment accuracy with a small amount of training data is recognized as a common challenge in current AI technology.

The technique described in Patent Document 1 was also conceived in response to such a problem. Patent Document 1 describes an apparatus that uses a modification stage classification model to determine whether or not the soil being transported needs to be modified. In addition, this modification stage classification model classifies soil into one of the modification stages, by learning multiple small area images cut out from each sample image prepared for each modification stage. It is something that is created.
Note that Patent Document 1 describes that fast Fourier transform is performed using the values of pixels (pixel values) constituting this small area image to obtain an amplitude spectrum. Then, the value of each pixel obtained after the fast Fourier transform is normalized, and the small area image after the transform is used for machine learning for learning and verification (Paragraph 0040 of the specification of Patent Document 1).

Furthermore, Patent Document 2 discloses that each time the captured image of the camera is updated due to an update of the signal input from the vehicle-mounted camera, a crosswalk is detected based on a spectrum pattern obtained by Fourier transforming a predetermined area of the captured image. A crosswalk detection device is described that includes an extraction means for extracting a zebra pattern of a corresponding predetermined period from a predetermined area.
Note that Patent Document 2 describes that it is determined whether or not a zebra pattern candidate corresponding to a crosswalk exists in an edge-processed image signal based on a comparison between the average value of the amplitude spectrum and an intensity threshold value. This is described (Paragraph 0042 of the specification of Patent Document 2).

Further, Patent Document 3 describes a detection device and a program aimed at improving the detection rate when detecting a detection target in an image. Patent Document 3 discloses that only objects (undetected objects) that are not detected by the first determination unit (first classification model) are determined by the second determination unit (second classification model), so that high It is stated that accuracy can be achieved (Paragraphs 0039 and 0040 of the specification of Patent Document 3).

Japanese Patent Application Publication No. 2020-041801 Japanese Patent Application Publication No. 2013-114652 Japanese Patent Application Publication No. 2021-157550

However, since the technology described in Patent Document 2 does not create data used for machine learning in the first place, the spectrum pattern obtained by the technology described in Patent Document 2 is sufficient data to produce effective learning results. It's not a thing.
Further, the technique described in Patent Document 3 aims to improve the determination accuracy by creating and using a plurality of classification models, so to speak. Therefore, (a lot of) training data and learning are required to create multiple classification models.

Note that the technology described in Patent Document 1 converts a captured image, but it can efficiently generate a large amount of learning data by cutting out multiple small area images from one image as learning data. It is something to be acquired. It is fine if the target for determination (soil) is similar no matter which part is cut out from the first image, such as soil, but if the image is a photographed image of mechanical parts such as screws or nuts, or a human body, If different objects appear in different parts of the image, it is not possible to obtain effective training data.

Therefore, the present invention provides an image processing device, an image processing method, and the like for creating data used in machine learning in order to achieve stable determination accuracy even with a small amount of data.

The image processing device of the present invention includes a first calculation means that numerically converts a photographed image into a plurality of color information for each pixel, and creates data by combining each of the numerically converted color information; and second calculation means for creating data used for machine learning by performing fast Fourier transform on the data created by the calculation means.
As a result, the object to be photographed in the photographed image is numerically converted into a plurality of pieces of color information and then combined to create integrated data, and then the combined data is further subjected to fast Fourier transform.

Further, the first calculation means scans the photographed image in the row direction and the column direction, numerically converts the pixels of the photographed image into a plurality of color information, and converts the pixels of the photographed image into a plurality of color information obtained by scanning in the row direction. It is preferable that the data be created by combining a plurality of pieces of color information obtained by scanning in the column direction.
As a result, the object to be photographed in the photographed image is scanned from a plurality of directions, including the row direction and the column direction, thereby acquiring and combining more pieces of color information.

Further, it is preferable that the first calculation means numerically converts the photographed image, which is a three-dimensional photographed image, into a plurality of color information for each volume element.
This allows more multiple color information to be acquired and combined.

The image processing system of the present invention includes the above-described image processing device, a light source that illuminates an object to be photographed, and a photographing means that photographs the object illuminated by the light source. Note that it is preferable that the object to be photographed is a mechanical part, and that there are a plurality of light sources that illuminate the object to be photographed from a plurality of different directions.
As a result, the object to be photographed is illuminated from a plurality of different directions by a plurality of light sources, and even if the object to be photographed is changed by the photographing means, a uniform photographed image without shadows can be obtained.

The image classification device of the present invention includes learning means for performing machine learning using teacher data created by the above-described image processing device.
Preferably, the learning means further performs machine learning using only normal data created by the above-mentioned image processing device based on a photographed image of a photographed object showing a normal state.
As a result, a trained model is generated that can classify the photographed object in the photographed image.

Further, the image processing device of the present invention numerically converts a photographed image into a plurality of color information for each pixel using the first calculation means, and creates data by combining the respective numerically converted color information. and a step of creating data used for machine learning by performing fast Fourier transform on the data created by the first calculation means, using the second calculation means.

Further, the image processing program of the present invention allows a computer to perform a first operation of numerically converting a photographed image into a plurality of color information for each pixel and creating data by combining the numerically converted color information. and a second calculation means that generates data used for machine learning by performing fast Fourier transform on the data created by the first calculation means.

(1) The image processing device of the present invention includes a first calculation means that numerically converts a photographed image into a plurality of color information for each pixel and creates data by combining each of the numerically converted color information. , and a second calculation means that creates data used for machine learning by performing fast Fourier transform on the data created by the first calculation means. Integrated data is created by numerically converting to color information and combining it, and then the integrated data is further subjected to fast Fourier transform, so it is possible to create feature quantities that are not affected by changes in the position or shape of anomalies existing in the photographed subject. It is possible to clearly extract data and create data that can be effectively used for machine learning.

(2) Further, the first calculation means scans the photographed image in the row direction and the column direction, numerically converts the pixels of the photographed image into a plurality of color information, and converts the pixels of the photographed image into a plurality of color information obtained by scanning in the row direction. The configuration creates data by combining color information and multiple pieces of color information obtained by scanning in the column direction. Since more color information is acquired and combined by scanning, it is possible to more clearly extract feature amounts that are independent of changes in the position or shape of an abnormal location present in the photographic subject.

(3) In addition, the first calculation means is configured to numerically convert a photographed image, which is a three-dimensional photographed image, into a plurality of color information for each volume element, so that even more plurality of color information can be acquired and combined. Therefore, it is possible to more clearly extract a feature quantity that is independent of changes in the position or shape of an abnormal part that exists in an object to be photographed.

(4) The image processing system of the present invention has the above-described image processing device, a light source that illuminates the object to be photographed, and a photographing means that photographs the object illuminated by the light source, and in particular, the object to be photographed is a machine. Because of the configuration, the object is illuminated from multiple different directions by multiple light sources, and even if the object changes depending on the shooting method, there will be no shadows. Since a uniform photographed image without any unevenness is obtained, it is possible to extract the feature amount of the photographed object stably without any unevenness.

(5) The image classification device of the present invention has a learning means that performs machine learning using teacher data created by the above-mentioned image processing device. With the configuration that performs machine learning using only the normal data created by the above-mentioned image processing device based on the photographed image, a trained model that can classify the photographed object in the photographed image is generated. , it is possible to classify the photographed object in the photographed image using the learned model.

Note that, according to the image processing device and the image processing program of the present invention, it is possible to achieve the same effects as the image processing device of the present invention.

1 is a schematic configuration diagram of an image processing system according to an embodiment of the present invention. FIG. 2 is a flow diagram for explaining an image processing method according to an embodiment of the present invention. 1 is a diagram for explaining an image processing method according to an embodiment of the present invention. FIG. These are diagrams for explaining photographed images, in which (A) is a photographed image in which a normal part is photographed, (B) is a photographic image in which an abnormal part (with rust) is photographed, and (C) is a photographic image in which an abnormal part (with rust) is photographed. This is a photographed image showing traces. 5 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on the photographed image shown in FIG. 4; FIG. (B) is a converted image created based on a photographed image showing an abnormal part (rust); (C) is a converted image created based on a photographed image showing an abnormal part (dents). This is a converted image created by FIG. 7 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on other images. FIG. 7 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on other images. FIG. 7 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on other images. FIG. 7 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on other images. FIG. 7 is a diagram for explaining a converted image created by the image processing method according to the embodiment of the present invention based on other images.

The embodiments of the present invention will be described in detail below, but the explanation of the constituent elements described below is an example (representative example) of the embodiments of the present invention, and the present invention will be described below without departing from the gist thereof. is not limited to the content.

[Image processing system]
FIG. 1 is a schematic configuration diagram of an image processing system according to an embodiment of the present invention. As shown in FIG. 1, an image processing system 1 according to an embodiment of the present invention includes an image processing device 10, a photographing means 20, and a light source 30.

The light source 30 illuminates the photographic object O, and is, for example, an LED light or a strobe light.

The photographing means 20 is for photographing the object O, and is, for example, a camera that photographs a two-dimensional image, or a camera equipped with a depth sensor that photographs a three-dimensional image that includes depth (depth) in addition to a two-dimensional (plane) image. A camera (3D camera), etc.

Photographed image data of the photographic object O illuminated by the light source 30 and photographed by the photographing means 20 is sent to the image processing device 10. At this time, the photographed image data can be sent from the photographing means 20 to the image processing device 10 via wired communication, wireless communication, or a storage medium such as a USB or an external hard disk.

[Image processing device]
The image processing device 10 processes captured image data to create a converted image. This converted image is used as data used in machine learning, such as teacher data.

Further, as shown in FIG. 1, the image processing device 10 includes an input means 110, a first calculation means 120, a second calculation means 130, a display means 140, a storage means 150, and an output means 160.

The input means 110 performs input processing of the photographed image photographed by the photographing means 20. As described above, the input means 110 receives the photographed image sent from the photographing means 20 via wired communication, wireless communication, or a storage medium, and stores it in the storage means 150.
Furthermore, the input means 110 accepts various input operations from the user of the image processing apparatus 10 via a keyboard, mouse, touch panel, or the like.

The storage unit 150 stores (saves) information necessary for the image processing device 10, and is, for example, a memory or a database. In addition to photographed images, the storage unit 150 stores converted images, images in the middle of conversion, programs for operating the image processing device 10, and the like.

The first calculation means 120 numerically converts the photographed image into a plurality of color information for each pixel, and creates data by combining the respective numerically converted color information.
When the captured image is a three-dimensional captured image, the first calculation means 120 numerically converts the captured image into a plurality of color information for each volume element, and combines the numerically converted color information to create data. Create.

The second calculation means 130 performs fast Fourier transform on the data created by the first calculation means 120 to create a transformed image. This converted image is used as data for machine learning.

The output means 160 outputs the converted image created by the second calculation means 130 to an external device or the like via wired communication, wireless communication, or a storage medium.

The display means 140 displays various information to the user, and is, for example, a display.

The image processing device 10 according to the embodiment of the present invention is configured by constructing (implementing) a program capable of realizing the above-described functions shown in FIG. 1 on, for example, a mobile PC, a desktop PC, or a tablet. It can be used as an image processing device.

[Image classification device]
Although not shown, as an embodiment of the present invention, there is an image classification device having a learning means for performing machine learning using a converted image created by the image processing device 10 as training data. As the teacher data, for example, as shown in FIGS. 4A, 4B, and 4C, a converted image created based on a photographed image showing a mechanical part (screw) is used as the teacher data.

In addition, the image classification device performs machine learning using learning means such as CNN (Convolution Neural Network), RNN (Recurrent Neural Network), AutoEncoder, or transfer learning to generate a trained model. .

Then, the image classification device determines the photographed image using the learned model. For example, the trained model is used to make inferences about photographic images of screws manufactured in a factory, and whether the screws in the photographic images are normal (no rust or dents) or abnormal (no rust or dents). Determine if there is rust or dents).

Note that the image processing apparatus 10 may include this learning means to provide the above-described determination function. In this case, the learning means performs learning using the transformed image stored in the storage means 150 as teacher data, and the generated trained model is stored in the storage means 150.

[Image processing method]
FIG. 2 is a flow diagram for explaining the image processing method according to the embodiment of the present invention.
Hereinafter, an image processing method using the image processing apparatus according to the present embodiment will be described with reference to each drawing. In this description, it is assumed that the image processing method according to the present embodiment is performed by the user.

First, a photographed image photographed by the photographing means 20 is input to the image processing device 10 (step S110). Then, input processing is performed by the input means 110, and the photographed image is stored in the storage means 150. Note that in this description, the photographed image will be described as a two-dimensional image.

Next, the first calculation means 120 numerically converts the photographed image into a plurality of color information for each pixel (step S120).
FIG. 3 is a diagram for explaining an image processing method according to an embodiment of the present invention. To make it easier to understand, a part of a photographed image is enlarged and explained. In the example shown in FIG. 3, the number of pixels of the part of the enlarged image is "8x8".

Then, the first calculation means 120 calculates a plurality of pieces of color information for each 8×8 pixel.
For example, if the multiple color information is red, green, and blue information based on the RGB color model, the color information of <red>, <green>, and <blue> for each 8×8 pixel is as follows. It is calculated as follows.

<Red>
255 175 72 32 210 216 53 8
202 158 99 66 199 195 49 24
119 110 120 96 156 139 105 95
100 141 114 112 114 86 217 224
69 115 104 112 146 110 244 247
62 85 103 129 122 116 73 66
112 94 63 108 154 175 35 0
140 107 32 78 181 208 43 0

<Green>
1 35 156 164 209 207 156 168
18 52 149 154 210 207 145 161
78 77 102 99 189 198 184 187
147 157 63 67 146 148 255 255
155 158 61 56 149 136 255 255
129 131 106 101 77 74 56 61
102 117 150 150 43 34 0 0
93 119 161 155 37 18 0 0

<Blue>
1 10 81 42 41 42 47 85
30 39 88 54 79 79 56 91
112 88 66 42 132 144 136 140
199 183 44 34 135 143 252 248
216 193 45 31 158 153 255 252
197 167 79 64 84 96 66 65
173 148 99 84 26 42 0 4
165 145 97 73 10 18 0 5

For example, the upper left number is the color information of the upper left pixel (the shaded area shown in FIG. 3) in the 8×8 pixels. Therefore, in this example, the <red> color information of the pixel in the diagonally shaded area shown in FIG. 3 is 255, the <green> color information is 1, and the <blue> color information is 1.

Then, the first calculation means 120 creates data by combining the numerically converted color information (step S130). In this explanation, the <green> color information for each pixel is combined after the <red> color information for each pixel of the photographed image, and the <blue> color information for each pixel is further followed. be combined.

For example, taking the color information in the 8x8 pixels as an example, when scanning a photographed image in the row direction (horizontal direction), this integrated data is
"255,175,72,......,208,43,0,......,1,35,156,......,18,0,0,......,1,10,81,...... ,18,0,5''
The data is a list of 64×3=192 pieces of color information, such as:

Then, the second calculation means 130 performs fast Fourier transform on the data created by the first calculation means to create a transformed image (step S140).

Finally, the output means 160 outputs the converted image (step S150).
The converted image obtained through the above procedure is used as training data in machine learning.

[Example 1]
FIG. 4 is a diagram for explaining a photographed image. As shown in FIGS. 4(A) to 4(C), the object to be photographed is a mechanical part (screw). Here, the photographed image shown in FIG. 4(B) shows a screw with rust (see the dotted circle), and the photographed image shown in FIG. 4(C) shows a dent (see the dotted circle). The picture shows a screw with a . On the other hand, the photographed image shown in FIG. 4A shows a normal screw without such rust or dents.

Further, FIG. 5 is a diagram for explaining a converted image created by the image processing method according to the present embodiment based on the photographed image shown in FIG. 4. Here, FIG. 5(A) is a converted image created based on the captured image shown in FIG. 4(A), and FIG. 5(B) is a converted image created based on the captured image shown in FIG. 4(B). , FIG. 5(C) is a converted image created based on the captured image shown in FIG. 4(C).

As shown in FIG. 5, the created converted image is an image in which frequency components are extracted as amplitudes (vertical axis). Note that since the color information does not have a time axis (time information), the second calculation means 130 performs fast Fourier transform using an arbitrary value (for example, a frequency of 100 Hz with a period of 10 milliseconds).

Here, when comparing FIG. 5(A) and FIG. 5(B), it can be seen that there are portions where the amplitudes are different (see the arrow portion in FIG. 5(B)). Furthermore, even when comparing FIG. 5(A) and FIG. 5(C), it can be seen that there are portions where the amplitudes differ (see the arrow portions in FIG. 5(C)). Note that even when comparing FIG. 5(B) and FIG. 5(C), it can be seen that the magnitudes of the amplitudes are different.

In other words, it can be said that the converted image shown in FIG. 5 has extracted features of the photographed object shown in the photographed image, such as rust, dents, and nothing. Therefore, this converted image can be said to be effective as training data that indicates normality or abnormality in machine learning, and by performing machine learning using this training data, stable judgment accuracy can be achieved even with a small amount of data. It can be said that it is possible to generate a trained model to obtain

[Example 2]
6 to 9 are diagrams for explaining converted images created by the image processing method according to the present embodiment based on other images. In FIGS. 6 to 9, the upper part shows an image assumed to be a photographed image. On the other hand, the bottom part shows a converted image created by the image processing method according to the present embodiment based on the image.

[Position change]
For example, the image shown at the top of FIG. 6A shows a six-pointed star pattern that can be assumed to be an abnormality (six-pointed star-shaped scratch). From this image, a converted image as shown at the bottom of FIG. 6(A) was created.

Further, the image shown at the top of FIG. 6(B) is obtained by changing only the position (location) of the hexagram pattern shown in the image shown at the top of FIG. 6(A). In other words, although the position of the hexagram pattern has changed, its size and color have not changed. From this image, a converted image as shown at the bottom of FIG. 6(B) was created.

Comparing the converted images in FIGS. 6(A) and 6(B), it can be seen that the magnitude and shape of the amplitude do not change. In other words, it can be seen that even if the location of scratches, rust, dents, etc. in the photographed image changes, the information (features) can be extracted accurately. In short, when a transformed image is used as training data in machine learning, it can be said to have high robustness against changes in the position of an abnormal part.

[Change color]
FIG. 7(A) is the same as the image and converted image shown in FIG. 6(A). It is described for comparison with FIG. 7(B).
On the other hand, FIG. 7(B) shows the hexagram pattern shown in the image shown at the top of FIG. 7(A) in a different color.

Comparing the converted images in FIGS. 7A and 7B, it can be seen that although the magnitude of the amplitude changes, the shape of the amplitude does not change. In other words, it can be seen that by correcting (for example, enlarging or reducing) the amplitude shown in the modified image, it is possible to extract features that can be determined to be an abnormal part.
Note that although the shape of the amplitude does not change, the magnitude of the amplitude changes, so it can be seen that it is possible to extract features that can be used to distinguish between abnormalities that differ only in color, such as red rust and blue rust.

[Change size]
Further, FIG. 8(A) is also the same as the image and converted image shown in FIG. 6(A). This is described for comparison with FIGS. 8(B) and (C).
On the other hand, Fig. 8(B) shows the hexagram pattern shown in the upper image of Fig. 8(A) changed in size (larger), and Fig. 8(C), conversely, is made smaller. This is what I did.

Comparing the converted images in FIGS. 8(A) to 8(C), it can be seen that the amplitude and shape of each image are different. Therefore, it can be seen that features that can distinguish the "degree of abnormality", such as the size of a scratch, can be extracted.

[shape change]
Further, FIG. 9(A) is also the same as the image and converted image shown in FIG. 6(A). This is described for comparison with FIGS. 9(B) and (C).
On the other hand, in FIG. 9(B), the shape of the pattern is changed from a six-pointed star to a hexagon, and in FIG. 9(C), the six-pointed star pattern is rotated 90 degrees to the left.

Comparing the transformed images in FIGS. 9(A) to 9(C), it can be seen that the amplitude and shape of each image are different. Therefore, it can be seen that features that can distinguish the "type of abnormality" such as scratches and dents, for example, can be extracted.

[Example 3]
FIG. 10 is a diagram for explaining a converted image created by the image processing method according to the present embodiment based on other images. In FIG. 10 as well, the upper part shows an image assumed to be a photographed image, and the lower part shows a converted image created based on the image by the image processing method according to the present embodiment.

For example, the image shown at the top of FIG. 10A shows a pattern of horizontal lines that can be assumed to be an abnormality (horizontal scratches). In addition, the image shown at the top of FIG. 10(B) has a pattern of vertical lines that can be assumed to be an abnormality (vertical line-like scratches), and the image shown at the top of FIG. A pattern of diagonal lines, which can be assumed to be diagonal scratches, is visible.
From these images, converted images as shown at the bottom of FIGS. 10(A) to 10(C) were created.

Comparing the converted images in FIGS. 10(A) and 10(B), it can be seen that the amplitude and shape of each image are different. On the other hand, it can be seen that the converted image in FIG. 10(C) is a combination of the amplitudes shown in the converted images in FIGS. 10(A) and 10(B).
In other words, the captured image is scanned not only in the row direction (horizontal direction) as described above, but also in the column direction (vertical direction), and data is created by combining the color information obtained from each scan. This makes it highly robust to changes in the shape of the abnormal part and the direction in which the abnormal part is provided (e.g., horizontal scratches, vertical scratches, diagonal scratches, etc.). I can say that.

Therefore, the first calculation means 120 scans the photographed image in the row and column directions, numerically converts the pixels of the photographed image into a plurality of color information, and converts the pixels of the photographed image into a plurality of color information obtained by scanning in the row direction. , data is created by combining a plurality of color information obtained by scanning in the column direction, and the data is fast Fourier transformed by the second calculation means 130 to create a transformed image, as shown in FIG. 9(A) The amplitude of the converted image based on the image shown above and the amplitude of the converted image based on the image shown above in FIG. 9(C) are considered to have approximately the same size and shape.

Although the embodiments of the present invention have been described as above, the described embodiments are merely examples, and the present invention is not limited to this content unless it departs from the gist thereof.

[Unsupervised learning]
For example, an image classification device has been described that has a learning means that performs machine learning using a converted image created by the image processing device 10 as training data. That is, the learning means has been described as performing machine learning based on teacher data including both normal data and abnormal data as described below.
- Normal data: Teacher data created based on a photographed image (FIG. 4(A)) of a photographic object (screw) showing a normal state.
- Abnormal data: Teacher data created based on photographed images (Fig. 4 (B), (C)) of photographed objects (screws) showing the above-mentioned conditions such as rust and dents.

In addition to supervised learning using supervised data labeled as ``normal,'' ``abnormal (rust),'' ``abnormal (dents),'' etc., the learning method can be unsupervised learning. You can also do it.
For example, the learning means can perform machine learning using only the normal data. Specifically, the learning means uses One Class SVM (Support Vector Machine), which is a method applied to one-class classification in unsupervised learning, to learn one class as normal data and determine the discrimination boundary. Outliers (abnormal data) can be detected using the identification boundary as a reference.

In other words, as is clear from the converted images shown in FIGS. 5(A) to 5(C), the image processing device 10 processes normal data (FIG. 5(A)) and abnormal data (FIGS. 5(B), ( C)) It has been possible to sufficiently extract the feature values that can determine the discrimination boundary between C) and C). Therefore, the learning means uses, for example, the data shown in FIG. 5(A) as a reference, and based on Euclidean distance, Mahalanobis distance, etc., the feature values indicated by amplitude size, shape, etc. are determined to be within the identification boundary. Learning can be performed such that data that is determined to be normal data is determined, and conversely, data that is determined to be outside the identification boundary is determined to be abnormal data.

As a practical problem, we would like to perform supervised learning that includes normal data and abnormal data, but it may be difficult to obtain this abnormal data. For example, in a factory that manufactures mechanical parts, the probability of producing a defective part is extremely low, ranging from 1 in 100,000 to 1 million. Therefore, it may be extremely difficult to obtain sufficient data on multiple types of abnormalities, such as red rust, blue rust, and dents.
According to the image processing device 10 and image classification device of the present invention, even in such a case, unsupervised learning can be performed using the created converted image, so stable judgment accuracy can be achieved even with a small amount of data. be able to.

In this way, the converted image created by the image processing device 10 can be used for supervised learning, and can also be used for unsupervised learning.

[Preprocessing]
Further, preprocessing (image processing) can also be performed on the photographed image.
For example, the first calculation means 120 converts the tones of the photographed image into red, green, and blue tones based on the RGB color model. In this case, since a plurality of images such as a photographed image converted to a red tone, a photographed image converted to a green tone, and a photographed image converted to a blue tone are generated, the first calculation means 120 further The generated (converted) image is numerically converted into a plurality of color information.

Conversion of tone based on the RGB color model means, for example, when the RGB value is specified from 0 to 255, in the case of a photographed image converted to a red tone, the red value is 255, and the green and blue values are 0. The purpose is to convert the image so that

Then, the first calculation means 120 numerically converts the converted image into a plurality of color information. At this time, (1) the first calculation means 120 may create data by combining a plurality of color information based on captured images that have been converted into red tones, green tones, and blue tones, respectively. . In addition, (2) data that combines multiple color information based on a captured image converted to a red tone, data that combines multiple color information based on a captured image converted to a green tone, and blue tone Data may be created by combining a plurality of pieces of color information based on captured images converted into .

By performing such pre-processing, it is possible to more clearly extract the feature amount of the abnormal location in the photographed image. In other words, it is possible to more accurately identify the abnormal location in the photographed image. This is because some abnormal areas have characteristic colors such as red rust or blue rust, so by performing such pretreatment, the characteristics of the abnormal area can be made more conspicuous.
For example, as described in (2) above, combined data is created based on the photographed image converted into a red tone by the first calculation means 120, and then the learning used for machine learning by the second calculation means 130. If training data is created based on captured images converted to green or blue tones, even if the training data cannot clearly extract the features of the abnormal location, the training data created based on the captured image converted to green or blue tone will be abnormal. In some cases, the feature values of a location can be clearly extracted.

Then, the image classification device performs machine learning using the learning data created based on the photographed images converted into red, green, and blue tones, and each of the learned models is generated. Then, when inference is performed using multiple trained models generated for a certain photographed image, if any one of them is determined to be abnormal, there is an abnormality in the photographed object in the photographed image. It can be determined that
In particular, in a detection method based on captured images using machine learning, it is permissible to some extent to determine normality as abnormality, but it is not permitted to determine abnormality as normality.

[Three-dimensional image]
In addition, in the above description, the photographed image was mainly described as a two-dimensional image, but the image processing device 10 creates a converted image based on a three-dimensional image that includes depth (depth) in addition to two-dimensional (plane) image. You can also do that. In this case, the first calculation means 120 numerically converts the captured image into a plurality of color information for each volume element, and combines the numerically converted color information to create data. Create. Then, the second calculation means 130 creates a transformed image by performing fast Fourier transform on the data.

As a result, color information is acquired based on three-dimensional information (XYZ), which has a larger amount of information than two-dimensional information (XY), so features of areas with abnormalities such as rust, dents, and scratches can be extracted. can do. For example, if there is a dent on the object to be photographed, the depth of the dent is darker than the front (shallow part) of the dent. In other words, there is a difference in color information, so the characteristics of the dent can be detected more accurately. can be extracted.

Furthermore, the imaging target may be not only mechanical parts such as screws and nuts, but also parts where an abnormality such as a malignant tumor occurs inside or outside the human body, such as the stomach or the skin.

In addition to red, green, and blue information based on the RGB color model, the plurality of color information may also be information on hue, saturation, and brightness based on the HSV model, or other color information. There may be.
The design of the color information can be changed as appropriate depending on the object to be imaged, such as whether the object is bright, dark, or glossy.

The present invention is effective in fields such as supervised learning and unsupervised learning as an image processing device and image processing method for creating data used in machine learning in order to achieve stable judgment accuracy even with small amounts of data. It is industrially useful because it can be used.

1 Image processing system 10 Image processing device 110 Input means 120 First calculation means 130 Second calculation means 140 Display means 150 Storage means 160 Output means 20 Photographing means 30 Light source O Photographing object

Claims (12)

a first calculation means that numerically converts the photographed image into a plurality of color information for each pixel and creates data by combining the respective numerically converted color information;
a second calculation means for creating data used for machine learning by fast Fourier transforming the data created by the first calculation means;
An image processing device having: The first calculation means scans the photographed image in the row direction and the column direction, numerically converts the pixels of the photographed image into a plurality of color information, and converts the plurality of color information obtained by scanning in the row direction. 2. The image processing apparatus according to claim 1, wherein the data is created by combining the color information and the plurality of color information obtained by scanning in the column direction. The image processing device according to claim 1, wherein the first calculation means numerically converts the photographed image, which is a three-dimensional photographed image, into a plurality of color information for each volume element. 4. The image processing apparatus according to claim 2, wherein the plurality of color information are red, green, and blue information based on an RGB color model, or hue, saturation, and lightness information based on an HSV model. The plurality of color information is red, green, and blue information based on an RGB color model,
The first calculation means converts the captured image into red, green, and blue tones based on an RGB color model, and numerically converts the converted image into the plurality of color information. The image processing device according to claim 2 or 3. A light source that illuminates the photographic subject,
Photographing means for photographing the photographing target illuminated by the light source;
An image processing device according to claim 4;
An image processing system with The object to be photographed is a mechanical part,
7. The image processing system according to claim 6, wherein there are a plurality of said light sources and illuminates said object to be photographed from a plurality of different directions. An image classification device comprising learning means for performing machine learning using teacher data created by the image processing device according to claim 1. The image classification according to claim 8, wherein the learning means further performs machine learning using only normal data created by the image processing device based on a photographed image of a photographed object showing a normal state. Device. A learned model generated by the image classification device according to claim 8 or 9. A step of numerically converting the photographed image into a plurality of color information for each pixel by the first calculation means, and creating data by combining the respective numerically converted color information;
creating data used for machine learning by performing fast Fourier transform on the data created by the first calculation means, using a second calculation means;
An image processing method comprising: computer,
a first calculation means that numerically converts the photographed image into a plurality of color information for each pixel and creates data by combining the respective numerically converted color information;
a second calculation means for creating data used for machine learning by fast Fourier transforming the data created by the first calculation means;
An image processing program that operates as an image processing device having.

Images