JP2021076711A - Scrap image capturing system, scrap image capturing method, image capturing support device, and program - Google Patents

Abstract

To provide a scrap image capturing system capable of generating a static image every time an iron scrap is lifted by a lifting tool.SOLUTION: A scrap image capturing system is provided, comprising a crane for lifting and transporting iron scraps using a lifting tool, detection means configured to detect it when the lifting tool reaches a given height, and a camera configured to photograph a range including the lifting tool when the lifting tool reaches the given height to generate a static image.SELECTED DRAWING: Figure 2

Description

The present invention relates to a scrap image capturing system, a scrap image capturing method, a photographing support device, and a program.

Generally, at a scrap receiving site of an electric furnace manufacturer, an inspector observes a place where iron scrap is lifted from a truck or a ship by a hanging tool such as a lifting magnet, transported, and dropped into a scrap yard, and the grade ratio of iron scrap is determined. ..

By the way, since the above-mentioned judgment by the inspector may vary depending on the experience of the inspector, the judgment is carried by a hanging tool so that it can be used not only as a record but also for re-judgment. It is required to save the image of the iron scrap.

However, whether the image of the iron scrap transported by the hanging tool is saved as a moving image or the image of the iron scrap taken regularly is saved as a still image, the image of the iron scrap subject to the judgment by the inspector is used. Difficult to extract. Further, in a moving image, blurring and blurring may occur, and the amount of data becomes excessive.

The present invention has been made in view of the above problems, and a main object thereof is a scrap image capturing system capable of generating a still image each time an iron scrap is lifted by a hanging tool, a scrap image capturing method, and the like. The purpose is to provide a shooting support device and a program.

In order to solve the above problems, the scrap imaging system according to one aspect of the present invention includes a crane that lifts and conveys iron scrap by a hanging tool, and a detection means that detects that the hanging tool has reached a predetermined height. And a camera that captures a range including the crane and generates a still image when the crane reaches a predetermined height.

Further, in the scrap imaging method of another aspect of the present invention, it is detected that the hanger of the crane that lifts and conveys the iron scrap by the hanger has reached a predetermined height, and the hanger reaches a predetermined height. When the height is reached, a still image is generated by photographing the range including the hanging tool with a camera.

Further, in the photographing support device of another aspect of the present invention, the detection means for detecting that the hanging tool has reached a predetermined height of the crane that lifts and conveys the iron scrap by the hanging tool, and the hanging tool are used. It is provided with an output means for outputting a shooting command to a camera that shoots a range including the crane and generates a still image when a predetermined height is reached.

Further, in the program of another aspect of the present invention, the detection means for detecting that the hanger has reached a predetermined height of the crane that lifts and conveys the iron scrap by the hanger, and the hanger are predetermined. When the height of the crane is reached, the computer functions as an output means for outputting a shooting command to a camera that shoots a range including the crane and generates a still image.

According to the present invention, it is possible to generate a still image each time the iron scrap is lifted by the hanging tool.

It is a figure which shows the example of a scrap receiving site. It is a figure which shows the configuration example of the scrap image taking system which concerns on embodiment. It is a figure which shows the example of a data set. It is a figure which shows the example of the image for learning. It is a flow chart which shows the procedure example of a learning phase. It is a flow chart which shows the procedure example of an inference phase. It is a figure which shows the example of the moving image and the estimation result taken by a video camera. It is a figure which shows the example of the still image taken by a camera. It is a figure which shows the example of the learning image which concerns on the modification. It is a flow chart which shows the procedure example of an inference phase. It is a figure which shows the configuration example of the scrap image taking system which concerns on the modification. It is a flow chart which shows the procedure example of an inference phase. It is a flow chart which shows the procedure example of a processing process. It is a figure for demonstrating the processing process. It is a figure for demonstrating the processing process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[System overview]
FIG. 1 is a diagram showing an example of a scrap receiving site where the scrap imaging system 100 according to the embodiment is installed. FIG. 2 is a block diagram showing a configuration example of the scrap image capturing system 100.

The scrap image capturing system 100 is a system for photographing the iron scrap S lifted by the hanger 91 of the crane 9 with the camera 3 at the scrap receiving site of an electric furnace maker or the like.

At the scrap receiving site, the iron scrap S is lifted from the truck T by the hanger 91 of the crane 9, transported, and lowered to the scrap yard Y. The iron scrap S is, for example, heavy scrap, and various sizes are mixed.

The crane 9 is provided with a hanging tool 91 such as a lifting magnet, and is operated by an operator A in the crane operating room R. The hanging tool 91 is not limited to the lifting magnet, and may be a grab bucket or the like.

At the scrap receiving site, a wall W is installed between the vehicle entry area where the truck T enters and the scrap yard Y. The hanger 91 holding the iron scrap S on the loading platform of the truck T rises to a height exceeding the wall W and moves to the scrap yard Y.

At the slap receiving site, a video camera 2 that photographs the hanging tool 91 to generate a moving image and a camera 3 (still camera) that photographs the hanging tool 91 to generate a still image are installed. The video camera 2 and the camera 3 are a part of the scrap image capturing system 100.

The video camera 2 is installed so as to include the loading platform of the truck T and the space above the loading platform T in the photographing range, and photographs the hanger 91 that lifts the iron scrap S from the loading platform of the truck T (see FIG. 7). Further, the video camera 2 is installed so as to include the upper ends of the scrap yard Y and the wall W in the shooting range.

The camera 3 is installed so as to capture a range including the hanger 91 located at a predetermined height. Specifically, the camera 3 is installed so as to include the hanging tool 91 located at a predetermined height and the iron scrap S lifted by the hanging tool 91 in the photographing range. The camera 3 magnifies a range narrower than the shooting range of the video camera 2 and shoots.

There may be a plurality of cameras 3. For example, a plurality of cameras 3 may be provided so as to photograph the iron scrap S lifted by the hanger 91 from different angles, or the camera 3 used according to the position of the hanger 91 in the width direction of the wall W. A plurality of cameras 3 may be provided so as to properly use.

It is preferable that the camera 3 photographs the iron scrap S lifted by the hanger 91 horizontally or from the bottom to the top so that the iron scrap S lifted by the hanger 91 can be photographed more. ..

In the crane operation room R, a display unit 4 for displaying an image taken by the camera 3 is installed. The driver A also serves as an inspector, and observes the actual iron scrap S lifted by the hanger 91 and the iron scrap S displayed on the display unit 4 to determine the grade ratio and the like.

In addition, a display unit 4 is also installed in a remote office, and one or more inspectors B in the office observe the iron scrap S displayed on the display unit 4 to determine the grade ratio and the like. May be good.

The scrap imaging system 100 may be applied to a scrap shipping site in a scrap processing company. At the scrap shipping site, the iron scrap S is lifted from the scrap yard Y by the hanger 91 of the crane 9, transported, and loaded on the truck T.

As shown in FIG. 2, the scrap image capturing system 100 includes a photographing support device 1, a video camera 2, a camera 3 (still camera), a database 5, and a learning device 6. These devices can communicate with each other via a communication network such as a LAN.

The shooting support device 1 includes a control unit 10. The control unit 10 is a computer including a CPU, a RAM, a ROM, a non-volatile memory, an input / output interface, and the like. The CPU of the control unit 10 executes information processing according to a program loaded into the RAM from the ROM or the non-volatile memory.

The control unit 10 includes an acquisition unit 11, a detection unit 12, and an output unit 13. These functional units are realized by the CPU of the control unit 10 executing information processing according to a program loaded from the ROM or the non-volatile memory into the RAM.

The program may be supplied via an information storage medium such as an optical disk or a memory card, or may be supplied via a communication network such as the Internet or a LAN.

The learning device 6 also includes a control unit 60 like the photographing support device 1. The control unit 60 includes an acquisition unit 61, a learning unit 62, and a storage unit 63. The learning device 6 may be composed of one or a plurality of server computers.

The photographing support device 1 and the learning device 6 can access the database 5. In the database 5, the trained model 51 constructed by the learning device 6 is readable and stored by the photographing support device 1.

The video camera 2 photographs the hanging tool 91 that conveys the iron scrap S, and outputs the generated moving image to the photographing support device 1. The camera 3 photographs the hanging tool 91 that conveys the iron scrap S in response to the photographing command from the photographing support device 1, and generates a still image.

[Learning phase]
FIG. 3 is a diagram showing an example of a data set used in the learning phase. FIG. 4 is a diagram showing an example of the learning image L included in the data set. The dataset used in the learning phase contains training images, labels, and ranges.

The “label” indicates that the object included in the range N in the learning image L is the hanger 91. The “range” represents a range N including the hanger 91 in the learning image L by coordinates. The "label" and "range" are teacher data, and are determined and given by the user, for example.

As shown in FIG. 4, the learning image L is, for example, an image of a hanger 91 for lifting iron scrap S. The iron scrap S lifted by the hanger 91 has a substantially inverted conical shape (a substantially inverted triangular shape in a side view) that gradually narrows toward the bottom. The learning image L may be subjected to a processing process described later. The learning image L may include an image of the hanger 91 in which the iron scrap S is not lifted.

Further, the learning image L may include an image taken by the camera 3 installed at the scrap receiving site, or may include an image taken by the video camera 2. Further, the learning image L may include an image of another hanging tool taken by another camera at another place.

FIG. 5 is a flow chart showing a procedure example of the learning phase realized in the learning device 6. The control unit 60 of the learning device 6 functions as the acquisition unit 61, the learning unit 62, and the storage unit 63 by executing the process shown in the figure according to the program.

First, the control unit 60 creates a large number of data sets including learning images, labels, and ranges (see S11, FIGS. 3 and 4).

Next, the control unit 60 acquires a part of the data set as training data (S12; processing as the acquisition unit 61).

Next, the control unit 60 executes machine learning using the acquired training data (S13; processing as the learning unit 62). Specifically, the control unit 60 constructs a trained model for estimating the range of the hanger from the image by using the learning image as input data, the label and the range as the teacher data, and the machine learning.

The trained model is an object detection model such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector) or Faster R-CNN.

Next, the control unit 60 acquires a part of the data set other than the training data as test data (S14), and evaluates the trained model using the acquired test data (S15). ).

After that, the control unit 60 stores the trained model whose evaluation is equal to or higher than a predetermined value in the database 5 (S16), and ends the learning phase.

[Inference phase]

FIG. 6 is a flow chart showing a procedure example of an inference phase as a scrap image photographing method according to an embodiment realized in the photographing support device 1. The control unit 10 of the shooting support device 1 functions as an acquisition unit 11, a detection unit 12, and an output unit 13 by executing the processes shown in the figure according to a program.

First, the control unit 10 acquires a moving image from the video camera 2 (S21; processing as the acquisition unit 11).

Next, the control unit 10 estimates the range of the hanger 91 in the moving image acquired from the video camera 2 using the trained model constructed in the learning phase (S22). Specifically, the control unit 10 sequentially inputs each of the plurality of still images (frames) included in the moving image into the trained model as input data.

Next, the control unit 10 determines whether or not the hanger 91 has reached a predetermined height based on the estimated range of the hanger 91 (S23; processing as the detection unit 12). Specifically, the control unit 10 determines whether or not the hanger 91 has risen to reach a predetermined height.

FIG. 7 is a diagram showing an example of a moving image V captured by the video camera 2. The moving image V shows a hanging tool 91 lifting the iron scrap S from the loading platform of the truck T, crossing the wall W, and transporting the iron scrap S to the scrap yard Y.

As the estimation result, the moving image V is provided with a range M surrounding the hanging tool 91 and a numerical value indicating the certainty of being the hanging tool 91. Further, the moving image V is also provided with a locus K of the center point of the range M surrounding the hanger 91.

A determination line L for determining whether or not the hanger 91 has reached a predetermined height is set in the moving image V. The determination line L is set at an arbitrary position by, for example, a user operation. In the illustrated example, the determination line L is set along the upper end of the wall W.

When the center point or the entire range M of the hanger 91 in the moving image V exceeds the determination line L from the lower side to the upper side, it is determined that the hanger 91 has reached a predetermined height.

Returning to the description of FIG. 6, when the hanger 91 reaches a predetermined height (S23: YES), the control unit 10 outputs a shooting command to the camera 3 (S24; processing as the output unit 13), and the camera The still image generated in 3 is saved (S25). Upon receiving the photographing command, the camera 3 photographs the hanging tool 91 and the iron scrap S lifted by the hanging tool 91 to generate a still image.

That is, when the hanger 91 crosses the determination line L from the lower side to the upper side because it goes from the loading platform of the truck T toward the scrap yard Y, the camera 3 takes a still image. On the other hand, when the hanger 91 crosses the determination line L from the upper side to the lower side because the hanger 91 goes from the scrap yard Y to the loading platform of the truck T, the camera 3 does not take a still image.

The timing at which the control unit 10 outputs a shooting command may be immediately after the hanger 91 reaches a predetermined height (the range M of the hanger 91 exceeds the determination line L), or the hanger 91 may be output. It may be after a predetermined time has elapsed since 91 reaches a predetermined height.

The control unit 10 repeats the processes S21 to S25 until all the iron scrap S has been conveyed from the loading platform of the truck T (S26). Whether or not all the iron scrap S has been transported is determined and input by, for example, the driver A or the like.

According to the embodiment described above, each time the hanger 91 lifts the iron scrap S, it is possible to obtain a plurality of still images taken by the camera 3.

The plurality of still images obtained in this way are attached to, for example, an acceptance slip that describes the determination results such as the grade ratio. As a result, the basis of the judgment in the acceptance inspection can be shown, and the reliability of the judgment result can be improved. In addition, by saving multiple still images, it is possible to review the basis of the judgment in acceptance inspection later, and it is also possible to use it as evidence when foreign matter or other non-acceptable items are mixed. It becomes.

FIG. 8 is a diagram showing an example of a still image C taken by the camera 3. The still image C taken by the camera 3 shows the hanging tool 91 and the iron scrap S lifted by the hanging tool 91. In the still image C, the range occupied by the hanger 91 and the iron scrap S lifted by the hanger 91 in the image is larger than that in the moving image V (see FIG. 7). Further, the still image C has a higher resolution than the moving image V, and has less blurring and blurring.

Not limited to this, if the video camera 2 is capable of shooting a high-resolution moving image, when the hanger 91 reaches a predetermined height from a plurality of still images (frames) included in the moving image. It may be configured to extract a still image of. That is, the video camera 2 may be the "camera" in the above aspect.

[First modification]
FIG. 9 is a diagram showing an example of a learning image used in the learning phase according to the first modification. FIG. 10 is a flow chart showing a procedure example of the inference phase according to the first modification. The configuration or procedure that overlaps with the above embodiment is given the same number, and detailed description thereof will be omitted.

As shown in FIG. 9, the data set used in the learning phase according to this example further includes the presence or absence of scrap in addition to the training image, label, and range. “Presence / absence of scrap” indicates whether or not the hanger 91 is lifting the iron scrap S.

The control unit 60 (see FIG. 2) of the learning device 6 uses the learning image as input data, the label, the range, and the presence / absence of scrap as teacher data, and estimates the range of the hanging tool and the presence / absence of lifting of iron scrap from the image. Build a trained model of.

For example, a softmax function is used for the output layer indicating whether or not the iron scrap is lifted in the trained model, and the accuracy of whether or not the iron scrap is lifted is output as a numerical value between 0 and 1.

As shown in FIG. 10, in the inference phase according to this example, the control unit 10 uses the trained model constructed in the learning phase, and the range and suspension of the hanger 91 in the moving image acquired from the video camera 2. It is estimated whether or not the iron scrap S is lifted by the tool 91 (S37).

Next, when the hanger 91 is lifting the iron scrap S (S38: YES) and the hanger 91 reaches a predetermined height (S23: YES), the control unit 10 takes a picture with the camera 3. The command is output (S24), and the still image generated by the camera 3 is saved (S25).

In this way, by detecting that the hanger 91 lifting the iron scrap S has reached a predetermined height and taking a picture with the camera 3, the hanger 91 not lifting the iron scrap S is taken with the camera 3. It is possible to prevent this from happening.

[Second modification]
FIG. 11 is a diagram showing a configuration example of the scrap image capturing system 100 according to the second modification. FIG. 12 is a flow chart showing a procedure example of the inference phase according to the second modification. The configuration or procedure that overlaps with the above embodiment is given the same number, and detailed description thereof will be omitted.

As shown in FIG. 11, the scrap image capturing system 100 according to this example further includes an adjusting unit 7 for adjusting the orientation of the camera 3. The adjusting unit 7 includes, for example, a motor having a plurality of axes. Further, the control unit 10 of the photographing support device 1 further includes a position calculation unit 14 that calculates the position of the hanging tool 91, and a direction adjustment unit 15 that drives the adjustment unit 7 to adjust the direction of the camera 3. There is.

As shown in FIG. 12, in the inference phase according to this example, when the hanger 91 reaches a predetermined height (S23: YES), the control unit 10 calculates the position of the hanger 91 based on the estimation result. (S47; processing as the position calculation unit 14), after adjusting the shooting direction of the camera 3 so as to face the hanger 91 (S48; processing as the direction adjusting unit 15), a shooting command is output to the camera 3. (S24).

By adjusting the shooting direction of the camera 3 so as to face the hanger 91 in this way, it is possible to prevent the hanger 91 and the iron scrap S lifted from the hanger 91 from being cut off in the still image.

[Third variant]
FIG. 13 is a flow chart showing an example of a procedure for processing the still image C generated by the camera 3. 14 and 15 are diagrams for explaining the processing. The control unit 10 of the photographing support device 1 functions as a processing means by executing the process shown in the figure according to a program.

First, the control unit 10 acquires the still image C from the camera 3 (S51). As shown in FIG. 14A, the iron scrap S included in the still image C has a substantially inverted triangular shape that gradually narrows downward from the hanger 91.

Next, the control unit 10 detects the range M of the hanger 91 in the still image C (S52), and the detection of the range M of the hanger 91 is estimated using the trained model as in the above S22. It will be realized.

Next, the control unit 10 cuts out a partial image CO defined from the still image C with reference to the range M of the hanger 91 (S53). As shown in FIG. 14B, the partial image CO is defined to include at least the area below the hanger 91.

Specifically, the left and right ends of the partial image CO are located slightly outside the left and right ends of the hanger 91. The width of the partial image CO is set to be slightly wider than the width of the hanger 91 (for example, about 1.1 to 1.5 times).

Further, the upper end of the partial image CO is set to the position of the upper end of the hanger 91. The lower end of the partial image CO is located below the lower end of the hanger 91 by a predetermined height. Specifically, the lower end of the partial image CO has a base having the same width as the partial image CO and overlaps with the lower end of the hanging portion 91, and is a range of an inverted triangle (for example, an inverted right triangle) that gradually narrows toward the bottom. It is the position of the lower end (apex) of the MN.

Next, the control unit 10 performs mask processing in which the range MN of the inverted triangle of the partial image CO is set as the display area and the other range MF is set as the non-display area (S54). The masking process is realized by painting the range MFs adjacent to the left and right of the range MN of the inverted triangle, for example, in black. Alternatively, the range MF may be transparent.

With the above, the processing process is completed. The image thus obtained is attached to the acceptance slip or stored as evidence instead of the still image C shown in FIG. According to this, it is possible to reduce the storage capacity of image data. Moreover, since the mask processing is applied, it is possible to exclude the background portion that does not need to be paid attention to.

The shape of the range MN used as the display area is not limited to an inverted triangle, and may be an inverted trapezoidal shape that gradually narrows toward the bottom, as shown in FIGS. 15 (c) and 15 (d), for example. As shown in FIGS. 15 (e) and 15 (f), it may have a rectangular shape. Further, as shown in FIGS. 15 (b), (d), and (f), the hanger 91 above the range MN to be the display area and both sides thereof may also be painted black as a non-display area.

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications can be made to those skilled in the art.

In the above embodiment, it is determined whether or not the hanger 91 has reached a predetermined height by using the trained model, but the present invention is not limited to this, and for example, an optical mark is attached to the outer peripheral surface of the hanger 91. By recognizing the optical mark in the moving image generated by the video camera 2, it may be determined whether or not the hanger 91 has reached a predetermined height.

Further, in the above embodiment, it is determined whether or not the hanger 91 has reached a predetermined height based on the moving image generated by the video camera 2, but the present invention is not limited to this, and for example, the crane 9 is equipped with a laser range finder. Etc. may be installed to determine whether or not the hanging tool 91 has reached a predetermined height.

1 shooting support device, 10 control unit, 11 acquisition unit, 12 detection unit, 13 output unit, 14 position calculation unit, 15 direction adjustment unit, 2 video camera, 3 camera, 4 display unit, 5 database, 51 trained model, 6 Learning device, 60 Control unit, 61 Acquisition unit, 62 Learning unit, 63 Preservation unit, 7 Adjustment unit, 9 Crane, 91 Suspension tool, 100 Scrap imaging system, Y scrap yard, S iron scrap, T truck, W wall , R Crane cab, A driver (acceptor), L learning image, N range, V moving image, M range, K locus, L judgment line, C still image

Claims (12)

A crane that lifts and transports iron scrap with a hanger,
A detection means for detecting that the hanger has reached a predetermined height, and
A camera that captures a range including the hanger and generates a still image when the hanger reaches a predetermined height.
Equipped with a scrap imaging system. The detection means detects that the hanger has reached a predetermined height based on a moving image generated by photographing the hanger.
The scrap imaging system according to claim 1. A video camera that captures the hanging tool and generates the moving image is further provided.
The scrap imaging system according to claim 2. The detection means estimates the range of the hanger in the moving image by using a trained model pre-constructed by machine learning using the range of the hanger in the learning image as teacher data.
The scrap imaging system according to claim 2 or 3. The detection means uses the trained model constructed by further using the presence or absence of lifting of the iron scrap as teacher data, further estimates the presence or absence of lifting of the iron scrap in the moving image, and lifts the iron scrap. Detects that the hanger has reached a predetermined height,
The scrap imaging system according to claim 4. The detection means detects that the hanger has reached a predetermined height when the hanger crosses the determination line set in the moving image.
The scrap imaging system according to any one of claims 2 to 5. Further comprising adjusting means for adjusting the orientation of the camera based on the position of the hanger in the moving image.
The scrap imaging system according to any one of claims 2 to 6. Further provided is a processing means for extracting a partial image from the still image including at least the lower range of the hanger, which is defined with reference to the range of the hanger.
The scrap imaging system according to any one of claims 1 to 7. The processing means performs mask processing in which a range of the partial image that gradually narrows downward from the hanger is used as a display area and the other range is set as a non-display area.
The scrap imaging system according to claim 8. The crane that lifts and transports iron scrap with a hanger detects that the hanger has reached a predetermined height, and detects it.
When the hanger reaches a predetermined height, a camera captures a range including the hanger to generate a still image.
Scrap image shooting method. A detection means for detecting that the hanger has reached a predetermined height of a crane that lifts and transports iron scrap with a hanger.
When the hanger reaches a predetermined height, an output means that shoots a range including the hanger and outputs a shooting command to a camera that generates a still image.
A shooting support device equipped with. A detection means for detecting that the hanger has reached a predetermined height of a crane that lifts and transports iron scrap with a hanger, and
An output means that outputs a shooting command to a camera that shoots a range including the hanging tool and generates a still image when the hanging tool reaches a predetermined height.
A program to make your computer work as.

Images