JP2021086285A - Sealed object detection system, sealed object detection method, estimation device, and program - Google Patents

Abstract

To provide a sealed object detection system capable of easily detecting a sealed object included in an aggregation of iron scraps.SOLUTION: A sealed object detection system includes: a camera for photographing an aggregation of iron scraps: an estimation unit for estimating whether or not a sealed object is included in the aggregation of iron scraps from a camera image generated by the camera using a learned model built in advance by machine learning with presence of the sealed object in a learning image as teacher data; and a notification unit for performing notification when it is estimated that the sealed object is included in the aggregation of iron scraps.SELECTED DRAWING: Figure 2

Description

The present invention relates to a hermetic detection system, a hermetic detection method, an estimation device, and a program.

Generally, a scrap processing company lifts iron scrap from a scrap yard with a crane and loads it on a truck or ship. In addition, electric furnace manufacturers use cranes to lift iron scrap from trucks or ships and unload it in the scrap yard.

Japanese Unexamined Patent Publication No. 7-286969

By the way, iron scrap accepted by electric furnace manufacturers is required to be free of sealed substances such as cylinders, fire extinguishers, jacks, and rolls.

The present invention has been made in view of the above problems, and a main object thereof is a sealed object detection system, a sealed object detection method, an estimation device, which can easily detect a sealed object contained in an aggregate of iron scrap. And to provide the program.

In order to solve the above problems, the sealed object detection system according to one aspect of the present invention was preliminarily constructed by machine learning using a camera that captures a set of iron scrap and the presence or absence of the sealed object in the learning image as teacher data. Using the trained model, it was estimated that the inference unit that estimates whether or not the set of iron scrap contains a sealed object from the camera image generated by the camera, and the set of iron scrap contains a closed object. It is provided with a notification unit for notifying a case.

Further, in another aspect of the sealed object detection method of the present invention, a trained model constructed in advance by machine learning is used by photographing a set of iron scrap with a camera and using the presence or absence of the sealed object in the training image as teacher data. From the camera image generated by the camera, it is estimated whether or not the set of iron scrap contains a sealed substance, and when it is estimated that the set of iron scrap contains a sealed substance, a notification is given.

In addition, the estimation device of another aspect of the present invention uses machine learning to acquire a camera image generated by a camera that captures a set of iron scraps and the presence or absence of a closed object in the learning image as teacher data. Using a trained model constructed in advance, an inference unit that estimates whether or not a sealed object is included in the set of iron scraps from the camera image, and an output unit that outputs an estimation result by the inference unit are provided.

Further, the program of another aspect of the present invention is preliminarily constructed by machine learning using a acquisition unit for acquiring a camera image generated by a camera for photographing a set of iron scraps and the presence or absence of a sealed object in the learning image as teacher data. Using the trained model, the computer functions as an inference unit that estimates whether or not the set of iron scrap contains a closed object from the camera image, and an output unit that outputs the estimation result by the inference unit. ..

According to the present invention, it becomes easy to detect a sealed substance contained in an aggregate of iron scrap.

It is a figure which shows the example of a scrap shipping / receiving site. It is a figure which shows the structural example of the closed matter detection system which concerns on embodiment. It is a figure which shows the example of a data set. It is a figure which shows the example of the image for learning. It is a flow chart which shows the procedure example of a learning phase. It is a flow chart which shows the procedure example of an inference phase. It is a figure which shows the output example of a display part. It is a flow diagram which shows another procedure example of an inference phase.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[System overview]
FIG. 1 is a diagram showing an example of a scrap shipping / receiving site in which the sealed object detection system 100 according to the embodiment is installed. FIG. 2 is a block diagram showing a configuration example of the sealed object detection system 100.

The sealed object detection system 100 is a system for detecting the sealed object E mixed in the iron scrap S at the scrap shipping / receiving site. The sealed object E is, for example, a cylinder, a fire extinguisher, a jack, a roll, or the like.

In the following, an example in which the sealed object detection system 100 is applied to a scrap shipping site of a scrap processing company will be described. At the scrap shipping site, the iron scrap S is lifted from the scrap yard Y by the crane 9 and loaded onto the truck T or the ship B.

Iron scraps S of various sizes are mixed in the scrap yard Y. The iron scrap S is, for example, heavy scrap.

The crane 9 includes a lifting magnet 91 and is operated by an operator A in the crane operation cab R. The crane 9 may be provided with a grab bucket or the like instead of the lifting magnet 91. The crane 9 may be a hydraulic excavator.

One or more cameras 2 are installed at the scrap shipping site. The camera 2 photographs the iron scrap S piled up in the scrap yard Y, the iron scrap S lifted by the crane 9, the iron scrap S loaded on the truck T or the ship B, and the like.

Further, at the scrap shipping site, a notification unit 3 including, for example, a rotating light and a siren for notifying the detection of the sealed object E is installed. Further, in the crane operation cab R, a display unit 4 for displaying a moving image taken by the camera 2 is installed.

The camera 2, the notification unit 3, and the display unit 4 installed at the scrap shipping site are a part of the sealed object detection system 100.

The sealed object detection system 100 may be applied to a scrap receiving site in an electric furnace manufacturer. At the scrap receiving site, the iron scrap S is lifted from the truck T or the ship B by the crane 9 and unloaded to the scrap yard Y.

As shown in FIG. 2, the sealed object detection system 100 includes an estimation device 1, a camera 2, a notification unit 3, a display unit 4, a database 5, and a learning device 6. These devices can communicate with each other via a communication network such as a LAN. Further, the estimation device 1, the database 5, and the learning device 6 may be installed in the cloud environment.

The estimation device 1 includes a control unit 10. The control unit 10 is a computer including a CPU, GPU, RAM, ROM, non-volatile memory, an input / output interface, and the like. The CPU or GPU of the control unit 10 executes information processing according to a program loaded into the RAM from the ROM or the non-volatile memory.

The control unit 10 includes an acquisition unit 11, an inference unit 12, and an output unit 13. These functional units are realized by the CPU or GPU of the control unit 10 executing information processing according to a program loaded from the ROM or the non-volatile memory into the RAM.

The program may be supplied via an information storage medium such as an optical disk or a memory card, or may be supplied via a communication network such as the Internet or a LAN.

The learning device 6 also includes a control unit 60 like the estimation device 1. The control unit 60 includes an acquisition unit 61, a learning unit 62, and a storage unit 63. The learning device 6 may be composed of one or a plurality of server computers, or may be installed in a cloud environment.

The estimation device 1 and the learning device 6 can access the database 5. In the database 5, the trained model 51 constructed by the learning device 6 is readable and stored by the estimation device 1.

The camera 2 photographs a set of iron scraps S and outputs the generated image to the estimation device 1. The notification unit 3 notifies the detection of the sealed object E in response to a command from the estimation device 1. The display unit 4 displays an image in response to a command from the estimation device 1.

As shown in FIG. 1, one of the cameras 2 is a camera 21 (hereinafter, referred to as “yard photographing camera 21”) that photographs a set of iron scraps S piled up in the scrap yard Y. The yard photographing camera 21 photographs the scrap yard Y from above to below.

The other camera 2 is a camera 22 (hereinafter referred to as “crane photographing camera 22”) that photographs a collection of iron scraps S lifted by the crane 9. It is preferable that the crane photographing camera 22 photographs the iron scrap S lifted by the crane 9 horizontally from the side or from the bottom to the top so that the iron scrap S lifted by the crane 9 can be photographed more.

Further, another camera 2 is a camera 23 (hereinafter, referred to as “loading camera 23”) that photographs a set of iron scraps S loaded on the truck T or the ship B. The loading camera 23 photographs the loading platform of the truck T or the ship B from above to below.

Here, a set of iron scraps S before being lifted by the crane 9 photographed by the yard photographing camera 21, a set of iron scraps S lifted by the crane 9 photographed by the crane photographing camera 22, and a loading photographing camera 23. The set of iron scraps S unloaded from the crane 9 photographed by the camera 9 is almost the same. That is, the yard photography camera 21, the crane photography camera 22, and the loading photography camera 23 observe substantially the same set of iron scrap S in different modes from each other.

In order to realize the sealed object detection system 100, it is preferable to provide at least a loading camera 23. In order to improve the estimation accuracy, it is preferable to combine one or both of the yard photographing camera 21 and the crane photographing camera 22.

[Learning phase]
FIG. 3 is a diagram showing an example of a data set used in the learning phase. FIG. 4 is a diagram showing an example of a learning image included in the data set. The dataset used in the learning phase contains training images, labels, and ranges.

The "label" indicates that the object included in the range N in the learning image L is the closed object E. Specifically, the "label" describes the type of sealed object such as a cylinder, a fire extinguisher, a jack, and a roll. The "range" represents the range N including the closed object E in the learning image L by coordinates. The "label" and "range" are determined and given by the user, for example.

4 (a) and 4 (b) are examples of the learning image L in which the sealed object E is included in the set of iron scrap S.

Of these, FIG. 4A shows a set of iron scraps S piled up in the scrap yard Y photographed by the yard photographing camera 21 or loaded onto a truck T or a ship B photographed by the loading photographing camera 23. It is a learning image L of a set of iron scrap S. FIG. 4B is a learning image L of a set of iron scraps S lifted by the crane 9 photographed by the crane photographing camera 22.

The learning image L is not limited to the image taken by the camera 2 installed at the scrap shipping site, and may be an image of a set of iron scraps S taken at another place.

Further, an image as shown in FIG. 4C, which does not contain the iron scrap S but contains the sealed object E alone, may be used as the learning image L.

FIG. 5 is a flow chart showing a procedure example of the learning phase realized in the learning device 6. The control unit 60 of the learning device 6 functions as the acquisition unit 61, the learning unit 62, and the storage unit 63 by executing the process shown in the figure according to the program.

First, the control unit 60 acquires a data set including a learning image, a label, and a range (see S11, FIGS. 3 and 4). The dataset is divided into training data, validation data, and test data (eg, 8: 1: 1).

Next, the control unit 60 acquires training data from the data set (S12; processing as the acquisition unit 61).

Next, the control unit 60 executes machine learning using the acquired training data (S13; processing as the learning unit 62). Specifically, the control unit 60 constructs a trained model for estimating the presence / absence and range of a closed object from a camera image using a learning image as input data and a label and a range as teacher data by machine learning.

The trained model is an object detection model such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector) or Faster R-CNN.

Next, the control unit 60 verifies the trained model with the verification data (S14), modifies the trained model so that the correct answer rate increases (S15), and repeats executing machine learning with the training data again. This cycle is repeated several hundred times.

Next, the control unit 60 acquires test data and evaluates the trained model using the acquired test data (S16).

After that, the control unit 60 stores the trained model whose evaluation is equal to or higher than a predetermined value in the database 5 (S17), and ends the learning phase.

[Inference phase]
FIG. 6 is a flow chart showing a procedure example of the inference phase as the closed object detection method according to the embodiment realized in the estimation device 1. The control unit 10 of the estimation device 1 functions as an acquisition unit 11, an inference unit 12, and an output unit 13 by executing the processes shown in the figure according to a program.

First, the control unit 10 acquires a camera image from the camera 2 (S21; processing as the acquisition unit 11). The camera image is, for example, a still image taken by the camera 2 at a predetermined timing. Not limited to this, the camera image may be, for example, one still image extracted from a plurality of still images (frames) included in the moving image taken by the camera 2.

Specifically, the control unit 10 executes the process shown in the figure each time it acquires a camera image from one or more of the yard shooting camera 21, the crane shooting camera 22, and the loading shooting camera 23 (see FIG. 1). To do.

In one or more of the yard photography camera 21, the crane photography camera 22, and the loading photography camera 23, a set of iron scrap S is lifted from the scrap yard Y by the crane 9, transported, and loaded onto the truck T or ship B1. At least one camera image is taken for each stroke and output to the control unit 10.

The yard photographing camera 21 is at the timing before the collection of iron scrap S is lifted from the scrap yard Y by the crane 9, and before the surface layer portion of the pile of iron scrap S piled up in the scrap yard Y (that is, before being lifted by the crane 9). A set of iron scrap S) is photographed.

The crane photographing camera 22 photographs the set of iron scrap S lifted by the crane 9 at the timing when the crane 9 that lifts the set of iron scrap S reaches a predetermined height.

The loading camera 23 is a surface layer portion (that is, a pile of iron scrap S) loaded on the truck T or ship B at a timing after the set of iron scrap S is unloaded on the truck T or ship B by the crane 9. , A collection of iron scrap S after being unloaded from the crane 9).

Next, the control unit 10 uses the trained model constructed in the learning phase, uses the camera image acquired from the camera 2 as input data, and determines the presence or absence of the sealed object E in the set of iron scrap S captured in the camera image. The range is estimated (S22; processing as the inference unit 12).

The estimation of the presence or absence of the sealed object E is realized by calculating a numerical value indicating the accuracy of the sealed object E. Then, when the numerical value indicating the accuracy of the sealed object E exceeds the threshold value, it is determined that the sealed object E is included. For example, a softmax function is used in the output layer indicating the presence or absence of the sealed object E in the trained model, and the accuracy of the sealed object E is output as a numerical value between 0 and 1.

When it is determined that the set of iron scrap S contains the sealed object E (S23: YES), the control unit 10 drives the notification unit 4 (see FIGS. 1 and 2) to alarm the detection of the sealed object E. (S24; processing as the output unit 13).

Further, the control unit 10 marks the estimated range of the sealed object E on the camera image, and outputs the camera image marked with the range of the sealed object E to the display unit 4 (see FIGS. 1 and 2) (S25 and). S26; processing as the output unit 13).

FIG. 7 is a diagram showing an output example of the display unit 4. The camera image C output to the display unit 4 is marked with a range M surrounding the sealed object E. Further, a numerical value indicating the accuracy of the sealed object E is also attached adjacent to the range M surrounding the sealed object E.

In the figure, the camera image C taken by the crane shooting camera 22 is illustrated, but the camera images taken by the yard shooting camera 21 and the loading shooting camera 23 are also the range M surrounding the sealed object E and the sealed object E. A numerical value indicating the accuracy of is attached.

According to the embodiment described above, by estimating the presence or absence of the sealed object E from the trained model, it becomes easy to detect the sealed object E contained in the set of iron scrap S. Further, by estimating the range of the sealed object E, it becomes easy to specify the position of the sealed object E in the aggregate of the iron scrap S.

Further, according to the embodiment, the camera image C (see FIG. 7) in which the range M of the sealed object E is marked is displayed on the display unit 4 (see FIG. 1) installed in the crane operation cab R. , The operator A of the crane 9 can easily identify the position of the sealed object E.

Further, according to the embodiment, based on a plurality of camera images obtained by photographing a set of substantially the same iron scrap S in different modes by the yard photography camera 21, the crane photography camera 22, and the loading photography camera 23 (see FIG. 1). Since the presence or absence of the sealed object E is estimated, it is possible to improve the detection accuracy of the sealed object E when a plurality of cameras 2 are used.

A plurality of yard photography cameras 21, crane photography cameras 22, and loading photography cameras 23 may be provided. In particular, if one crane photographing camera 22 is used, only one side can be seen, so it is effective to use a plurality of crane photographing cameras 22.

[Modification example]
FIG. 8 is a flow chart showing another procedure example of the inference phase. The process shown in the figure is executed for each stroke in which the set of iron scrap S is lifted from the scrap yard Y by the crane 9, transported, and loaded on the truck T or the ship B.

First, the control unit 10 acquires a camera image of a set of iron scrap S before being lifted by the crane 9 photographed by the yard photography camera 21 (S31; processing as the acquisition unit 11), and the iron captured in the camera image. The presence / absence and range of the sealed object E in the set of scrap S are estimated (S32; processing as the inference unit 12).

Further, the control unit 10 acquires a camera image of a set of iron scrap S lifted by the crane 9 photographed by the crane photographing camera 22 (S34; processing as the acquisition unit 11), and the iron scrap captured in the camera image. The presence / absence and range of the sealed object E in the set of S are estimated (S35; processing as the inference unit 12).

Further, the control unit 10 acquires a camera image of the set of iron scrap S after being unloaded from the crane 9 photographed by the loading camera 23 (S37; processing as the acquisition unit 11), and is reflected in the camera image. The presence / absence and range of the sealed object E in the set of the iron scrap S are estimated (S38; processing as the inference unit 12).

As a result of each estimation, when the numerical value indicating the accuracy of the sealed object E is equal to or greater than the threshold value (S33, S36, S39; YES), the control unit 10 notifies the sealed object E by an alarm as in the above embodiment. The camera image marked with the range of (S24 to S26; processing as the output unit 13) is output.

On the other hand, when the numerical value indicating the accuracy of the sealed object E is less than the threshold value (S39; NO), the control unit 10 is a plurality of cameras photographed by the yard photographing camera 21, the crane photographing camera 22, and the loading photographing camera 23. The total accuracy of each sealed object E in the image is calculated, and it is determined whether or not the total is equal to or greater than a predetermined value. When a plurality of yard photography cameras 21, crane photography cameras 22, and loading photography cameras 23 are provided, the threshold value is appropriately adjusted.

As a result of the determination, when the total accuracy of the sealed object E is equal to or higher than a predetermined value (S40; YES), the control unit 10 is a camera that marks the range of the sealed object E together with the notification by the alarm, as in the above embodiment. The image is output (S24 to S26; processing as the output unit 13).

On the other hand, when the total accuracy of the sealed object E is less than a predetermined value (S40; NO), the control unit 10 ends the process.

According to this, even if it is not estimated that there is a sealed object E in each of the plurality of camera images taken by the yard photographing camera 21, the crane photographing camera 22, and the loading photographing camera 23, the accuracy of the sealed object E is as high as possible. By performing the determination based on the total, it is possible to suppress the detection omission of the sealed object E.

The total accuracy of the sealed object E is not limited to the simple sum of the accuracy of the sealed object E, but may be the sum of the accuracy of the sealed object E after multiplying it by a predetermined weight.

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications can be made to those skilled in the art.

1 Estimator, 10 Control, 11 Acquisition, 12 Inference, 13 Output, 2 (21-23) Camera, 3 Notification, 4 Display, 5 Database, 51 Learned Model, 6 Learning, 60 Control Department, 61 Acquisition part, 62 Learning part, 63 Preservation part, 9 Crane, 91 Lifting magnet, 100 Sealed object detection system, Y scrap yard, T truck, B ship, R crane cab, A driver, S iron scrap, E Sealed object, L Learning image, N range, C camera image, M range

Claims (14)

A camera that shoots a collection of iron scrap,
Using a trained model constructed in advance by machine learning using the presence or absence of a sealed object in the training image as teacher data, it is estimated from the camera image generated by the camera whether or not the sealed object is included in the set of iron scraps. Inference part to do
A notification unit that notifies when it is estimated that the set of iron scrap contains a sealed object,
A sealed object detection system. In the trained model, the range of the sealed object in the training image is further constructed as teacher data.
The inference unit further estimates the range of the sealed object in the camera image.
The sealed object detection system according to claim 1. A display unit for displaying the camera image marking the range of the sealed object is further provided.
The sealed object detection system according to claim 2. A plurality of the cameras provided at different positions for photographing a collection of the iron scraps.
The inference unit estimates whether or not the sealed object is included in the set of iron scrap from the plurality of camera images generated by the plurality of cameras.
The sealed object detection system according to any one of claims 1 to 3. Further equipped with a crane to lift some iron scrap from the piled iron scrap,
The sealed object detection system according to any one of claims 1 to 4. The inference unit estimates from the camera image of the iron scrap set lifted by the crane whether or not the sealed object is included in the iron scrap set lifted by the crane.
The sealed object detection system according to claim 5. The inference unit estimates from the camera image of the set of iron scrap before being lifted by the crane whether or not the sealed object is included in the set of iron scrap before being lifted by the crane.
The sealed object detection system according to claim 5 or 6. The inference unit estimates from the camera image of the set of iron scraps unloaded from the crane whether or not the sealed object is included in the set of iron scraps unloaded from the crane.
The sealed object detection system according to any one of claims 5 to 7. The reasoning unit includes the camera image of the set of iron scrap lifted by the crane, the camera image of the set of iron scrap before being lifted by the crane, and the unloaded iron scrap from the crane. Based on two or more of the camera images of the iron scrap set, it is estimated whether or not the iron scrap set contains the sealed object.
The sealed object detection system according to any one of claims 5 to 8. At least a part of the learning image is an image in which the sealed object is included in a set of iron scrap.
The sealed object detection system according to any one of claims 1 to 9. At least a part of the learning image is an image in which the sealed object is contained alone.
The sealed object detection system according to any one of claims 1 to 10. Take a picture of a set of iron scrap with a camera
Using a trained model constructed in advance by machine learning using the presence or absence of a sealed object in the training image as teacher data, it is estimated from the camera image generated by the camera whether or not the sealed object is included in the set of iron scraps. And
Notify when it is presumed that the collection of iron scrap contains a hermetically sealed material.
Sealed object detection method. An acquisition unit that acquires a camera image generated by a camera that shoots a set of iron scrap,
Using a trained model constructed in advance by machine learning using the presence or absence of a closed object in the training image as teacher data, an inference unit that estimates whether or not the set of iron scrap contains a closed object from the camera image, and an inference unit.
An output unit that outputs the estimation result by the inference unit and
An estimation device. Acquisition unit, which acquires a camera image generated by a camera that shoots a set of iron scrap,
An inference unit that estimates whether or not the set of iron scrap contains a closed object from the camera image using a trained model constructed in advance by machine learning using the presence or absence of a closed object in the training image as teacher data, and an inference unit. ,
An output unit that outputs the estimation result by the inference unit,
A program that makes a computer work as.

Images