JP2021163078A - Foreign matter detection device, foreign matter removal device, and foreign matter detection method - Google Patents

Abstract

To provide a foreign matter detection device capable of accurately detecting foreign matter from iron-based scrap mixed with foreign matter.SOLUTION: A foreign matter detection device 1 is provided, comprising a target image acquisition unit 2 configured to acquire a captured target image of a target object including iron-based scrap, and an image discrimination unit 3 configured to detect foreign matter by discriminating iron-based scrap and foreign matter other than the iron-based scrap from the target object on the basis of the target image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a foreign matter detecting device, and more particularly to a foreign matter detecting device for detecting foreign matter mixed in iron-based scrap.
The present invention also relates to a foreign matter removing device that removes foreign matter by using such a foreign matter detecting device.
Furthermore, the present invention also relates to a foreign matter detecting method for detecting foreign matter mixed in iron-based scrap.

In order to manufacture steel products while reducing the environmental load, it is desired to increase the use of iron-based scrap as an iron raw material. Iron-based scrap collected from the city is produced as waste of various products such as processing equipment, automobiles, and electric equipment. These products are composed of various devices such as electronic devices used for control and the like, motors serving as actuators, various sensors, and electric wires connecting electronic devices with actuators and sensors. Iron-based scrap is obtained by extracting iron-based materials from the waste of such products, but iron-based materials and other materials are often closely connected during product manufacturing, so other than iron. Materials containing components may be mixed in iron scrap. Materials containing components other than iron include, for example, electric motors containing copper, electric wires containing copper, electronic substrates containing copper or other metals, empty cans that may contain liquids, and the like.

On the other hand, when producing steel products, the composition constituting the steel products is important. For example, if a steel product contains a large amount of copper as a foreign substance, it causes cracks during manufacturing, which causes a problem. When this copper component is melted together with iron scrap for manufacturing steel products, it is difficult to remove it in the manufacturing process. Regarding foreign substances, there may be problems other than copper. For example, when a beverage can containing a liquid inside is mixed with iron scrap, the liquid may vaporize due to high temperature and the beverage can may burst and damage the manufacturing equipment when the iron scrap is melted. There is.

As described above, it is desirable to remove the material containing components other than iron as much as possible before melting because it causes various problems in the production of steel products when melted together with iron-based scrap.
In order to remove non-iron materials that become foreign substances from iron-based scrap, it is necessary to identify the position of the foreign substances in the iron-based scrap.

In recent years, an object recognition technique based on image segmentation using so-called deep learning, in which the type of an imaged object is determined for each image element (pixel) from an image obtained by capturing an object, has been developed. For example, in the semantic segmentation that distinguishes a road from an automobile, if the road is determined to be "0" and the automobile is determined to be "1", when an image of the road and the automobile is input, the vehicle is located on the road. The image element is determined to be "0", the image element located above the automobile is determined to be "1", and the result matrix has the same size as the input image and is represented by two values of "0" and "1". Is output. By extracting the coordinates of the value "1" in the result matrix, the position of the automobile can be specified.

Machine learning using a large amount of teacher data is required in order to perform highly accurate type determination for each image element by image segmentation. Each teacher data has a teacher image that captures an object as an object, and a teacher label with a label that can identify the object on the image element in which the object to be discriminated in the teacher image is captured. Is.

In particular, iron-based scrap is not produced by a dimensionally controlled manufacturing process, so it has various shapes and sizes, and its posture is random. Foreign substances containing components other than iron also have various shapes and sizes, and their postures are random. There are almost infinite combinations of images in which foreign matter with various shapes, sizes, and postures is mixed with iron-based scraps with various shapes, sizes, and postures. Must prepare a large number of teacher images and teacher labels containing foreign matter in iron scrap.

Here, the method of Patent Document 1 has been proposed as a method of creating a large amount of teacher data including a teacher image and a teacher label. According to the method of Patent Document 1, a first machine learning model of image segmentation is generated by machine learning using initial teacher data consisting of a certain amount of teacher images and teacher labels prepared in advance, and the first machine learning is performed. Teacher labels for these teacher images are created by image segmentation of teacher images other than the teacher images in the initial teacher data using the model. In this way, a large number of teacher images and teacher labels are created.

Japanese Unexamined Patent Publication No. 2019-101535

However, even if a large amount of teacher data is created by the method of Patent Document 1 and an attempt is made to identify the position of a foreign substance from an image obtained by capturing an iron-based scrap containing a foreign substance by using image segmentation, a first machine learning model is generated. There is a problem that it is difficult to prepare a certain amount of teacher images and teacher labels necessary for this purpose in a quantity sufficient to distinguish foreign substances from the image of iron-based scrap containing foreign substances.

The reason is that most of the foreign matter mixed in the iron-based scrap is removed by the iron-based scrap supplier, so there are few images of the iron-based scrap mixed with the foreign matter. Furthermore, since the shapes, sizes, and orientations of iron scrap and foreign matter are almost random, it is difficult to prepare an image assuming all mixed patterns.

In addition, even if a large amount of images of iron-based scrap containing foreign matter can be obtained, it is difficult to distinguish between foreign matter and iron-based scrap because they are often similar to each other, and many skilled workers. It is difficult to secure. Even if a large number of skilled workers are gathered, the work of finding foreign substances from the teacher image in which iron scrap and foreign substances with strong randomness in shape, size, and posture are mixed and creating a teacher label is manually performed. It takes a lot of time and effort to do it.

In the first place, although it is possible to photograph foreign matter removed from iron-based scrap, it is rare to photograph iron-based scrap in a state where foreign matter is mixed, so it is often difficult to secure a teacher image itself. ..
For this reason, conventionally, it has been difficult to accurately detect and remove foreign matter from iron-based scrap in which foreign matter is mixed.

The present invention has been made to solve such a conventional problem, and an object of the present invention is to provide a foreign matter detecting device capable of accurately detecting foreign matter from iron-based scrap in which foreign matter is mixed.
Another object of the present invention is to provide a foreign matter removing device for removing foreign matter from iron-based scrap in which foreign matter is mixed by using such a foreign matter detecting device.
Another object of the present invention is to provide a foreign matter detecting method capable of accurately detecting foreign matter from iron-based scrap in which foreign matter is mixed.

The foreign matter detection device according to the present invention includes an object image acquisition unit that acquires an image of an object in which an object including iron scrap is imaged, and an object other than iron scrap and iron scrap from the object based on the object image. It is provided with an image discrimination unit that detects foreign matter by discriminating it from foreign matter.

The image discrimination unit divides the object image into a plurality of image elements, determines whether each image element is an iron-based scrap or a foreign substance other than the iron-based scrap, and determines whether the image element is a foreign substance. It is preferable to include a type determination unit that recognizes the aggregate as a foreign substance.
The type determination unit is composed of a multi-layer neural network, and the parameters of the multi-layer neural network are a teacher image in which foreign matter is mixed in iron-based scrap and a predetermined marker at a position corresponding to the teacher image and where the foreign matter exists in the teacher image. It is preferable that machine learning is performed based on a plurality of teacher data including a teacher label labeled by.

The image discrimination unit is a type that uses a teacher data creation unit that creates a plurality of teacher data and a plurality of teacher data created by the teacher data creation unit to machine-learn the parameters of the multi-layer neural network that constitutes the type determination unit. It is preferable to include a determination function learning unit.
The teacher data creation unit is composed of an iron-based scrap image acquisition unit that acquires an iron-based scrap image in which only iron-based scrap is captured, a foreign matter-containing image acquisition unit that acquires a foreign matter-containing image containing foreign matter, and a foreign matter-containing image. An image cut-out part that cuts out an image part of a foreign object, a teacher image creation part that creates a teacher image by pasting an image part of a foreign substance that is cut out by the image cut-out part and has an iron-based scrap as a background on an iron-based scrap image. It is preferable to include a teacher label creation unit that creates a teacher label corresponding to the teacher image.

The foreign matter removing device according to the present invention includes the above-mentioned foreign matter detecting device, a foreign matter position specifying unit that identifies the position of the foreign matter in the real space based on the determination result by the type determination unit, and the foreign matter identified by the foreign matter position specifying unit. It is provided with a foreign matter removing portion that removes foreign matter from the object based on the position.

The foreign matter detection method according to the present invention includes an object image acquisition step of acquiring an object image in which an object including iron scrap is imaged, and iron scrap and foreign matter other than iron scrap based on the object image. It is a method including a discrimination step of detecting a foreign substance by discriminating.

According to the present invention, the object image acquisition unit acquires an object image in which an object including iron scrap is imaged, and the image discrimination unit acquires iron scrap and iron from the object based on the object image. Since the foreign matter is detected by distinguishing it from foreign matter other than the scrap, it is possible to accurately detect the foreign matter from the iron scrap in which the foreign matter is mixed.

It is a block diagram which shows the structure of the foreign matter detection apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows the object image which imaged the object containing iron scrap and foreign matter. It is a figure which shows the object image divided into a plurality of image elements. It is a figure which displayed the determination matrix obtained by performing the type determination for each image element with respect to the object image of FIG. 3 in accordance with the position of an image element. It is a block diagram which shows the structure of the teacher data creation part used for the foreign matter detection apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the iron scrap image. It is a figure which shows the foreign matter containing image. It is a figure which shows the image part of the foreign matter cut out from the foreign matter containing image. It is a figure which shows the teacher image created by pasting the image part of a foreign substance on an iron scrap image. It is a figure which shows the teacher image divided into a plurality of image elements. It is a figure which shows the teacher label created corresponding to the teacher image of FIG. It is a block diagram which shows the structure of the foreign matter removing apparatus which concerns on Embodiment 2. It is a figure which shows typically the foreign matter removing part used in the foreign matter removing apparatus. It is a flowchart which shows the operation which removes foreign matter from the object containing iron scrap and foreign matter, and charges iron scrap into an electric furnace. It is a figure which shows typically the foreign matter removing part which removes a foreign matter.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
Embodiment 1
FIG. 1 shows the configuration of the foreign matter detecting device 1 according to the first embodiment.
The foreign matter detection device 1 is connected to an object image acquisition unit 2 that acquires an object image by imaging an object containing iron-based scrap and an object image acquisition unit 2 by wire or wirelessly, and is an object image. It is provided with an image discrimination unit 3 for discriminating between iron-based scrap and foreign matter other than iron-based scrap from the object based on the above.

Here, iron-based scrap can be procured from, for example, a scrap supplier. Iron-based scrap collected from the market by scrap companies is produced as waste for automobiles, building materials, etc., and waste made of materials other than iron is likely to be mixed. Among the materials other than iron mixed in iron-based scrap, there are materials that are not desirable as steel products and those that cause problems in the production process. Examples of steel products containing undesired components include electric motors, electric wires, and electronic substrates containing copper, and those that cause problems in the production process are beverage cans containing liquid inside and explosive inside. It is a gas pipe or the like in which the gas raw material of the above remains. If the steel sheet contains copper, it may cause cracks during processing of the steel sheet. If a beverage can containing a liquid or a gas pipe containing an explosive gas raw material is charged into an electric furnace together with iron scrap, an explosion may occur in the electric furnace and damage the electric furnace and surrounding manufacturing equipment. ..

The object image acquisition unit 2 has a function of transferring the acquired object image to the image discrimination unit 3 by wire or wirelessly. As the object image acquisition unit 2, not only a general visible light camera that uses visible light to image an object, but also an infrared camera that uses infrared light to image an object can be used. Further, the object image acquisition unit 2 can be configured by one camera, or can be configured by a plurality of cameras. Since the object image can only capture the surface of the object as seen from the camera, for example, cameras are installed in each of a plurality of manufacturing processes of steel products, and the object image acquisition unit 2 composed of these plurality of cameras is used. Therefore, it is desirable to acquire a plurality of object images.

As shown in FIG. 1, the image discrimination unit 3 includes a type determination unit 4 connected to the object image acquisition unit 2, a type determination function learning unit 5 connected to the type determination unit 4, and a type determination function learning. It has a teacher data creation unit 6 connected to the unit 5.

The type determination unit 4 divides the object image acquired by the object image acquisition unit 2 into a plurality of image elements, and determines whether each image element is an iron-based scrap or a foreign substance other than the iron-based scrap. Then, a determination matrix composed of a plurality of matrix elements having the same number and the same arrangement as the plurality of image elements of the object image is output. A numerical value capable of discriminating between iron scrap and foreign matter is input to each matrix element of the determination matrix. That is, a numerical value serving as a predetermined marker is input to the determination matrix as a matrix element corresponding to the image element determined to be a foreign substance, and the matrix element corresponding to the image element determined to be a foreign substance is described above. A numerical value different from the numerical value that serves as a predetermined indicator of is input. Therefore, it is possible to detect the foreign matter mixed in the iron-based scrap based on the numerical value of each matrix element of the determination matrix output from the type determination unit 4.
Such a type determination unit 4 is composed of a multi-layer neural network composed of a large number of perceptrons.

The type determination function learning unit 5 and the teacher data creation unit 6 are sequentially connected to the type determination unit 4, and the parameters of the multi-layer neural network constituting the type determination unit 4 are created by the teacher data creation unit 6. Machine learning is performed by the type determination function learning unit 5 using a plurality of teacher data.
FIG. 2 shows an example of the object image GA acquired by the object image acquisition unit 2. The object image GA is an image obtained by capturing an object A including the iron scrap S, and the object image GA shown in FIG. 2 is an object in which foreign matter F is mixed between the iron scrap S. Object A is in the picture.

The type determination unit 4 is composed of a multi-layer neural network composed of a large number of machine-learned perceptrons, and as shown in FIG. 3, after dividing the object image GA into a plurality of image elements E, each image element E It is determined whether it is iron-based scrap S or foreign matter F, and a determination matrix LA representing the determination result as shown in FIG. 4 is output. The determination matrix LA is a matrix composed of a plurality of matrix elements having the same number and the same arrangement as the plurality of image elements E of the object image GA.

The determination matrix LA shown in FIG. 4 corresponds to the object image GA shown in FIG. 3, and corresponds to the image element E determined to be a foreign object F in the object image GA, and is shown by a thick line in FIG. A numerical value "1" serving as a predetermined marker is input as the two matrix element MFs represented by, and a numerical value "0" is input to the matrix element corresponding to the image element E determined not to be a foreign object F. Has been done.
Therefore, from the determination matrix LA shown in FIG. 4, the foreign matter F is reflected in the image element E of the object image GA corresponding to the two matrix element MFs in which the numerical value "1" is input, and the numerical value "0" is It means that the foreign matter F does not appear in the image element E of the object image GA corresponding to the other input matrix elements, and it is determined that the foreign matter F corresponds to the two matrix elements MF of the numerical value "1". The aggregate of the two image elements E can be recognized as the foreign matter F.

As shown in FIG. 1, the type determination function learning unit 5 is connected to the type determination unit 4, and the teacher data creation unit 6 is connected to the type determination function learning unit 5.
The parameters of the multi-layer neural network constituting the type determination unit 4 are machine-learned by the type determination function learning unit 5, so that the type determination unit 4 is an iron scrap for each image element E of such an object image GA. It is possible to determine whether it is S or foreign matter F.
In order for the machine-learned type determination unit 4 to perform type determination with high accuracy, it is desired that the type determination function learning unit 5 machine-learns the type determination unit 4 using a large amount of teacher data.

The teacher data creation unit 6 connected to the type determination function learning unit 5 can create a large number of teacher data. The large number of teacher data created by the teacher data creation unit 6 is composed of a teacher image and a teacher label corresponding to the teacher image, respectively.

The teacher image is an image in which foreign matter F is mixed with iron scrap S. The resolution of the teacher image is limited as long as the iron scrap S and the foreign matter F can be distinguished, and the size of the teacher image is not limited. The teacher image may be a color image or a black-and-white image, but in consideration of the possibility of identifying the foreign matter F from the iron-based scrap S due to the difference in color, the teacher image is to improve the accuracy of the type determination by the type determination unit 4. The image is preferably a color image.

On the other hand, the teacher label corresponds to the teacher image and is labeled with a predetermined label at the position where the foreign matter F exists in the teacher image. Specifically, the teacher label is a matrix composed of a plurality of matrix elements having the same number and the same arrangement as the plurality of image elements E of the teacher image when the teacher image is divided into a plurality of image elements E. , A numerical value "1" which is a predetermined marker representing the foreign matter F is input to the matrix element corresponding to the image element E in which the foreign matter F exists in the teacher image, and corresponds to the other image element E in which the foreign matter F does not exist. It is composed of a matrix in which a numerical value "0" is input to the matrix element to be used.

As shown in FIG. 5, the teacher data creation unit 6 includes an iron-based scrap image acquisition unit 7, a foreign matter-containing image acquisition unit 8, an image cutting-out unit 9 connected to the foreign matter-containing image acquisition unit 8, and an iron-based image. It has a teacher image creation unit 10 connected to both the scrap image acquisition unit 7 and the image cutout unit 9, and a teacher label creation unit 11 connected to the teacher image creation unit 10.

As shown in FIG. 6, the iron-based scrap image acquisition unit 7 acquires the iron-based scrap image G1 in which only the iron-based scrap S is imaged, and the iron-based scrap image acquisition unit 7 acquires only the iron-based scrap S by imaging the iron-based scrap S. It may be configured by a camera that acquires the image G1, or may acquire the image signal of the iron scrap image G1 in which only the iron scrap S is captured via a network, communication, or the like.
Since there are iron-based scraps S having various shapes, it is desirable to acquire a large number of images of iron-based scraps S having various shapes.

As shown in FIG. 7, the foreign matter-containing image acquisition unit 8 acquires the foreign matter-containing image G2 in which the foreign matter F is contained in at least a part thereof, and captures the foreign matter F in the same manner as the iron-based scrap image acquisition unit 7. It can also be configured by a camera that acquires the foreign matter-containing image G2 by the above, or may acquire the image signal of the foreign matter-containing image G2 in which the foreign matter F is captured via a network, communication, or the like.

Since the teacher image created by the teacher image creation unit 10 described later is obtained by pasting the image portion of the foreign matter F cut out from the foreign matter-containing image G2 on the iron-based scrap image G1, the background is generated when the teacher image is generated. For the purpose of reducing the difference from the iron-based scrap image G1, it is desirable to image the foreign matter F against the background of the iron-based scrap S, for example, as shown in FIG.
Further, since there are various shapes of the foreign matter F mixed in the iron-based scrap S when producing a steel product, it is desirable to acquire a large number of images of the foreign matter F having various shapes.

As shown in FIG. 8, the image cutting unit 9 cuts out the image portion G3 of the foreign matter F from the foreign matter-containing image G2 acquired by the foreign matter-containing image acquisition unit 8. Specifically, the image cutting unit 9 removes the foreign matter-containing image G2 of the portion other than the foreign matter F according to the shape of the foreign matter F by image processing on the foreign matter-containing image G2, and cuts out the image portion G3 of the foreign matter F. .. For example, the rectangular portion B of the broken line shown in FIG. 7 can be set so as to be in contact with the upper end, the lower end, the left end, and the right end of the foreign matter F, and the foreign matter F can be cut out from the foreign matter-containing image G2 along the rectangular portion B. .. If the image portion G3 of the foreign matter F can be cut out so as not to include the background of the foreign matter F as much as possible, the image processing method itself for cutting out is not limited.

The teacher image creation unit 10 attaches the image portion G3 of the foreign matter F cut out by the image cutout unit 9 onto the iron scrap image G1 acquired by the iron scrap image acquisition unit 7, so that FIG. 9 shows. Create a teacher image GT as shown. Since the image portion G3 of the foreign matter F is attached to the iron-based scrap image G1, the teacher image GT is created as an image in which the foreign matter F is mixed with the iron-based scrap S.

The teacher label creating unit 11 divides the teacher image GT into a plurality of image elements E as shown in FIG. 10, and then, as shown in FIG. 11, has the same number as the plurality of image elements E of the teacher image GT. Moreover, the teacher label LT is created as a matrix composed of a plurality of matrix elements having the same array. At this time, a numerical value "1" which is a predetermined marker representing the foreign matter F is input to the matrix element MF corresponding to the image element E in which the foreign matter F exists in the teacher image GT, and other images in which the foreign matter F does not exist. A numerical value "0" is input to the matrix element corresponding to the element E.

Here, since the teacher image GT is created by pasting the image portion G3 of the foreign matter F cut out from the foreign matter-containing image G2 on the iron-based scrap image G1, the teacher label creating unit 11 is created from the teacher image GT. The teacher label LT can be easily and mechanically created by simply inputting the numerical value "1" into the matrix element corresponding to the position where the image portion G3 of the foreign matter F is pasted without taking the trouble of identifying the foreign matter F. be able to.
In FIG. 11, for convenience, the numerical value "1" is input to the matrix element MF corresponding to the foreign matter F, and the numerical value "0" is input to the other matrix elements, but the matrix element MF corresponding to the foreign matter F is input. If and other matrix elements have different numerical values, they can be other numerical values.

Further, the teacher image creation unit 10 is cut out from a plurality of iron-based scrap images G1 acquired by the iron-based scrap image acquisition unit 7 and a plurality of foreign matter-containing images G2 acquired by the foreign matter-containing image acquisition unit 8. By randomly combining the image portion G3 of the foreign matter F, and further randomly setting the position, angle, size of the foreign matter F, inversion of the foreign matter F, and the number of foreign matter F in one teacher image GT. A large number of teacher image GTs can be created without actually preparing an image in which foreign matter F is mixed in the iron-based scrap S.
The teacher label creation unit 11 creates a large number of teacher label LTs corresponding to the large number of teacher image GTs created by the teacher image creation unit 10 in this way.

The type determination function learning unit 5 constitutes the type determination unit 4 by using a large number of teacher image GTs and a large number of teacher label LTs created by the teacher image creation unit 10 and the teacher label creation unit 11 of the teacher data creation unit 6. By machine learning the parameters of the multi-layer neural network, it is possible to improve the accuracy of the type determination by the type determination unit 4.

The image discrimination unit 3 of the foreign matter detection device 1 including the type determination unit 4, the type determination function learning unit 5, and the teacher data creation unit 6 as described above includes a CPU (central processing unit) and a DRAM connected to the CPU. By a main storage device such as dynamic random access memory), a large-capacity storage device such as SSD (solid state drive) or HDD (hard disk drive) connected to the CPU, and a program that operates the CPU. Can be configured.
A large number of teacher data created by the teacher data creation unit 6 and used as a model for the type determination unit 4 to perform type determination, and parameters of the multi-layer neural network constituting the type determination unit 4 are stored in a large-capacity storage device. , The CPU performs the calculation while using the main storage device, so that the type determination process by the type determination unit 4 is executed.

Next, the operation of the foreign matter detecting device 1 according to the first embodiment will be described with reference to FIGS. 1 to 11.
First, the object image acquisition unit 2 takes an image of the object A including the iron-based scrap S collected for manufacturing the steel product, and acquires the object image GA as shown in FIG.
The acquired object image GA is transferred from the object image acquisition unit 2 to the image discrimination unit 3 by wire or wirelessly, and is input to the type determination unit 4 of the image discrimination unit 3.

The type determination unit 4 is composed of a multi-layer neural network composed of a large number of perceptrons machine-learned using a large number of teacher data, and the object image GA input to the type determination unit 4 is shown in FIG. As described above, after being divided into a plurality of image elements E, it is determined by image segmentation whether the image element E is an iron-based scrap S or a foreign matter F. Then, as shown in FIG. 4, the numerical value "1" is input to the matrix element MF corresponding to the image element E determined to be the foreign matter F, and corresponds to the image element E determined not to be the foreign matter F. The determination matrix LA in which the numerical value "0" is input to the matrix element is output from the type determination unit 4.
By extracting the matrix element MF in which the numerical value "1" is input from the determination matrix LA, it is possible to detect that the object A contains the foreign matter F. Further, the aggregate of the image elements E corresponding to the matrix element MF of the numerical value "1" and determined to be the foreign matter F can be recognized as the foreign matter F.

In the type determination unit 4, the type of iron scrap S or foreign matter F is determined for each image element E by image segmentation. Generally, the parameters of the multi-layer neural network are adjusted by machine learning. Is said to require a number of teacher data that is about 10 times or more the number of parameters. In order to prepare such a large number of teacher data, when trying to collect a large number of teacher images by photographing iron-based scrap in which real foreign substances are mixed, various iron-based scraps and foreign substances having different shapes and sizes are used. It is necessary to collect these iron-based scraps and foreign substances by mixing them in various mixing methods and to image their respective states.

However, since the iron-based scrap and the foreign matter have a weight of several tens of kg to several thousand kg, a great deal of labor is required to move the iron-based scrap and the foreign matter. Therefore, it is not realistic to actually collect the teacher image necessary for machine learning of the parameters of the multi-layer neural network.
Even if it is possible to acquire an image of iron-based scrap mixed with real foreign matter, in order to generate a teacher label corresponding to the teacher image, the worker himself / herself uses the foreign matter mixed in the iron-based scrap. It is necessary to identify the position of the foreign substance by visual judgment and label the position with a numerical value indicating that the foreign substance is a foreign substance. It is not easy for the operator to judge the foreign matter mixed in the iron scrap for a large number of teacher images.

In the foreign matter detecting device according to the first embodiment, the teacher image GT is formed by pasting the image portion G3 of the foreign matter F cut out from the foreign matter-containing image G2 containing the foreign matter F at least in part on the iron-based scrap image G1. Is created, and the teacher label LT is created by inputting the numerical value "1" into the matrix element corresponding to the position where the image portion G3 of the foreign matter F is pasted. For this reason, a large number of teacher image GTs in which various foreign substances F having different shapes, sizes, and numbers are mixed in the iron scrap S by various mixing methods are used without photographing the iron scrap in which the actual foreign substances are mixed. A large number of teacher label LTs corresponding to these teacher image GTs can be easily and mechanically created.

In general, when the operation of pasting the second image on the first image is performed, if the positional relationship of the second image with respect to the first image is different from the actual one, the pasting is extremely unnatural. It may be an attached image. However, the iron-based scrap S and the foreign matter F, which are the targets of the type determination in the present invention, have a large randomness in shape, size, and orientation, and the image portion G3 of the foreign matter F is randomly arranged on the iron-based scrap image G1. Also, it is possible to generate an image close to the actual product.

In particular, if the foreign matter-containing image G2 in which the foreign matter F is imaged is acquired against the background of the iron-based scrap S and the image portion G3 of the foreign matter F is cut out from the foreign matter-containing image G2, the iron-based scrap is formed on the image portion G3 of the foreign matter F. Even if the background composed of S remains, the background is easily assimilated into the iron-based scrap image G1, and a high-quality teacher image GT can be obtained. By performing machine learning using a large number of high-quality teacher image GTs and a large number of teacher labels LT created in this way, it is possible to significantly enhance the type determination ability of the type determination unit 4.

Embodiment 2
FIG. 12 shows the configuration of the foreign matter removing device 21 according to the second embodiment. The foreign matter removing device 21 removes the foreign matter F mixed in the iron-based scrap S by using the foreign matter detecting device 1 of the first embodiment.
The foreign matter removing device 21 includes the foreign matter detecting device 1 described in the first embodiment, the foreign matter positioning unit 22 connected to the type determination unit 4 of the foreign matter detecting device 1, and the foreign matter removing unit 22 connected to the foreign matter positioning unit 22. It includes a control unit 23 and a foreign matter removing unit 24 connected to the foreign matter removing control unit 23.

The foreign matter position specifying unit 22 specifies the position of the foreign matter F mixed in the iron-based scrap S in the real space based on the determination result by the type determination unit 4, and controls the position information of the foreign matter F in the real space to remove the foreign matter. Output to unit 23.
Since the type determination unit 4 performs type determination for each image element E by image segmentation and outputs a determination matrix LA representing the determination result, the foreign matter position identification unit 22 is in the determination matrix LA output from the type determination unit 4. By checking the matrix element MF corresponding to the foreign matter F and recognizing the aggregate of the image elements E determined to be the foreign matter F as the foreign matter F, the entire position of the foreign matter F on the object image GA is specified. be able to. Further, since the object image GA is an image obtained by capturing the object A including the iron-based scrap S by the object image acquisition unit 2, the foreign matter position specifying unit 22 is the entire foreign matter F on the object image GA. Based on the position, the position of the foreign matter F including the external coordinates of the foreign matter F in the real space can be specified.

The foreign matter removal control unit 23 removes the foreign matter F from the object A in which the foreign matter F is mixed in the iron scrap S based on the position information of the foreign matter F in the real space output from the foreign matter position specifying unit 22. The operation of the foreign matter removing unit 24 is controlled.
The foreign matter removing unit 24 is an actuator that operates to remove the foreign matter F under the control of the foreign matter removing control unit 23, and as shown in FIG. 13, with respect to the object A including the iron-based scrap S. The grip portion 25 is held so as to be movable in the three-dimensional direction including the X direction, the Y direction, and the Z direction. Note that FIG. 13 shows an object image acquisition unit 2 installed in the vicinity of the object A.

By using the foreign matter removing device 21 according to the second embodiment, the foreign matter F can be removed from the object A including the iron scrap S and the foreign matter F.
FIG. 14 shows a flowchart of a method of charging the iron-based scrap S into the electric furnace after removing the foreign matter F from the object A containing the iron-based scrap S and the foreign matter F.
The operation of charging the iron-based scrap S from which the foreign matter F has been removed into the electric furnace will be described with reference to FIGS. 14, 3-4, 12-13, and 15.

First, in step S1, the object A containing the iron-based scrap S is transported to the steel product manufacturing facility.
Next, in step S2, as shown in FIG. 13, the object A is imaged by the object image acquisition unit 2 installed in the vicinity of the object A, and the object image GA is acquired. Note that FIG. 13 shows the object A in which the iron-based scrap S is mixed with the foreign matter F.
Further, the object image GA acquired by the object image acquisition unit 2 is transferred from the object image acquisition unit 2 to the type determination unit 4 by wire or wirelessly.

Further, in step S3, as shown in FIG. 3, the object image GA is divided into a plurality of image elements E by the type determination unit 4, and then iron-based scrap S and foreign matter are added to each image element E by image segmentation. Which of F is determined. Then, as shown in FIG. 4, the numerical value "1" is input to the matrix element MF corresponding to the image element E determined to be the foreign matter F, and corresponds to the image element E determined not to be the foreign matter F. The determination matrix LA in which the numerical value "0" is input to the matrix element is output from the type determination unit 4.

When the determination matrix LA is output from the type determination unit 4, in step S4, the foreign matter F from the object A is detected based on the determination matrix LA.
When the foreign matter F is detected from the object A in step S4, the process proceeds to step S5, and the foreign matter position specifying unit 22 mixes the foreign matter F with the iron-based scrap S based on the determination matrix LA output from the type determination unit 4. The position of the foreign matter F in the real space is specified, and the position information of the foreign matter F in the real space is output.

Further, in step S6, the foreign matter removal control unit 23 controls the operation of the foreign matter removal unit 24 based on the position information of the foreign matter F output from the foreign matter position identification unit 22, and the foreign matter F is removed from the object A. At this time, as shown in FIG. 15, the grip portion 25 of the foreign matter removing portion 24 is moved to the position of the foreign matter F in the real space, and the foreign matter F is removed from the object A in a state of being gripped by the grip portion 25. NS.
After the foreign matter F is removed from the object A in this way, in step S7, only the iron-based scrap S is charged into an electric furnace (not shown) of the manufacturing facility.
If the foreign matter F is not detected from the object A in step S4, it is determined that the object A is composed only of the iron-based scrap S, and the process proceeds from step S4 to step S7, and the object A is composed of only the iron-based scrap S. The object A is charged into the electric furnace.

By removing the foreign matter F from the object A, molten iron using only iron-based scrap S as a raw material can be performed in the electric furnace. Therefore, for example, the cracking of the steel product due to the mixing of the copper component is prevented, and the electric furnace is damaged due to the charging of the beverage can containing the liquid and the gas pipe containing the explosive gas raw material together with the iron scrap S. It becomes possible to manufacture high quality steel products by preventing the above.

13 and 15 show a foreign matter removing portion 24 that moves in a three-dimensional direction while the grip portion 25 is suspended, but the present invention is not limited to this. For example, an actuator such as a so-called robot arm in which a grip portion is attached to the tip of an arm that can move in a three-dimensional direction can be used as the foreign matter removing portion 24.
Further, when removing the foreign matter F containing iron, a foreign matter removing portion 24 that attracts the foreign matter F by an electromagnet may be used.

The object image acquisition unit 2 for acquiring the object image GA may be installed at a position where the object A transported to the steel product manufacturing facility can be imaged, and the object may be captured. Since the object image GA is acquired without contacting A, the degree of freedom in installing the object image acquisition unit 2 is high. Therefore, the object image GA can be acquired by the object image acquisition unit 2 without obstructing the flow line of the manufacturing site and without lowering the productivity.

1 Foreign matter detection device, 2 Object image acquisition unit, 3 Image discrimination unit, 4 Type determination unit, 5 Type determination function learning unit, 6 Teacher data creation unit, 7 Iron scrap image acquisition unit, 8 Foreign matter-containing image acquisition unit, 9 Image cropping unit, 10 Teacher image creation unit, 11 Teacher label creation unit, 21 Foreign matter removal device, 22 Foreign matter position identification unit, 23 Foreign matter removal control unit, 24 Foreign matter removal unit, 25 Grip part, S Iron scrap, F Foreign matter , A object, GA object image, E image element, LA judgment matrix, MF matrix element, G1 iron scrap image, G2 foreign matter containing image, G3 foreign matter image part, B rectangular part, GT teacher image, LT teacher label ..

Claims (7)

An object image acquisition unit that acquires an object image in which an object including iron scrap is captured,
A foreign matter detecting device including an image discriminating unit that detects a foreign matter by discriminating between the iron-based scrap and a foreign matter other than the iron-based scrap from the object based on the object image. The image discrimination unit divides the object image into a plurality of image elements, determines whether the image element is an iron-based scrap or a foreign substance other than the iron-based scrap, and determines that the object is a foreign substance. The foreign matter detection device according to claim 1, further comprising a type determination unit that recognizes the aggregate of the image elements as the foreign matter. The type determination unit is composed of a multi-layer neural network.
The parameters of the multi-layer neural network are a teacher image in which foreign matter is mixed in iron-based scrap and a teacher label corresponding to the teacher image and labeled at a position in the teacher image where the foreign matter is present with a predetermined marker. The foreign matter detection device according to claim 2, wherein machine learning is performed based on a plurality of teacher data including. The image discrimination unit
The teacher data creation unit that creates the plurality of teacher data, and
The foreign matter according to claim 3, which includes a type determination function learning unit that machine-learns parameters of the multilayer neural network constituting the type determination unit using the plurality of teacher data created by the teacher data creation unit. Detection device. The teacher data creation department
An iron-based scrap image acquisition unit that acquires an iron-based scrap image in which only the iron-based scrap is imaged, and an iron-based scrap image acquisition unit.
A foreign matter-containing image acquisition unit that acquires a foreign matter-containing image containing the foreign matter,
An image cutout portion that cuts out an image portion of the foreign matter from the foreign matter-containing image,
A teacher image creating unit that creates the teacher image by pasting an image portion of the foreign substance cut out by the image cutting unit and having the iron scrap as a background on the iron scrap image.
The foreign matter detection device according to claim 4, further comprising a teacher label creating unit that creates the teacher label corresponding to the teacher image. The foreign matter detection device according to any one of claims 1 to 5,
A foreign matter position specifying unit that specifies the position of the foreign matter in the real space based on the determination result by the type determination unit,
A foreign matter removing device including a foreign matter removing portion for removing the foreign matter from the object based on the position of the foreign matter specified by the foreign matter positioning portion. An object image acquisition step for acquiring an object image in which an object including iron scrap is captured, and
A method for detecting a foreign matter, which comprises a discrimination step of detecting the foreign matter by discriminating between the iron-based scrap and the foreign matter other than the iron-based scrap from the object based on the object image.

Images