CN115279684A - Control system, information processing device, and information processing method - Google Patents

Abstract

Time lost in the roll-up of the crane is reduced. A control system (100) includes: a photographing device (7) for photographing a crane (93) transporting garbage; and an information processing device (1) for inputting an image obtained by the above-mentioned photographing into a state determination model constructed by machine learning The output value detects when the grab bucket ( 934 ) can be rolled up; and a control device ( 3 ) performs a winding operation when detecting that the grab bucket ( 934 ) can be rolled up.

Description

Control system, information processing device, and information processing method

technical field

The present invention relates to a control system etc. which control the operation of a crane which transports garbage.

Background technique

Currently, cranes are used in facilities such as garbage incineration facilities to transport garbage. And it is also developing the technology to automatically control this kind of crane. For example, the following Patent Document 1 discloses a technique for adjusting the amount of garbage picked up by a crane bucket in order to obtain a good combustion state.

prior art literature

patent documents

Patent document 1: Japanese laid-open patent publication "JP-A-2002-295821"

Contents of the invention

(1) Technical problems to be solved

The purpose of the above-mentioned prior art is to obtain a good combustion state, and there is room for improvement from the viewpoint of reducing the time loss at the time of automatic control of the crane. Specifically, in the prior art described above, it is difficult to optimize the timing at which the grab bucket is rolled up after grabbing garbage with the grab bucket. In addition, such a timing is generally determined by the operator of the crane while visually checking the conditions of the grab bucket and the garbage.

Here, the above-mentioned Patent Document 1 describes that the opening and closing control of the grab bucket is performed according to the measurement time of the timer. However, when it is configured to use timer control to carry out the control of rolling up the grab bucket after a predetermined time has elapsed from the start of the action of picking up garbage, in order to avoid rolling up before it becomes a state that can be rolled up, it has to be rolled up. The prescribed time is set to be longer. And in such a structure, there exists a problem that time loss occurs until the grab bucket can be rolled up until the grab bucket is rolled up.

An object of one aspect of the present invention is to realize a control system or the like capable of reducing the time loss from the start of rolling up to the time of rolling up, compared with the case of rolling up by control using a timer.

(2) Technical solutions

In order to solve the above-mentioned technical problems, the control system of one aspect of the present invention includes: a photographing device that photographs a series of states of a crane that carries garbage after grabbing garbage with a grab bucket and rolling up the grab bucket in the state of grabbing garbage. An information processing device that detects that the grab bucket can be rolled up based on an output value obtained by inputting the image obtained by the above-mentioned photographing into a state judgment model, wherein the state judgment model is obtained by making the grab The image of the grab bucket in the state where the garbage can be rolled up is constructed as the machine learning of the teacher data; and the control device, when the information processing device detects that the grab bucket can be rolled up, makes the crane Carry out the above-mentioned rolling action of grab bucket.

In addition, in order to solve the above-mentioned technical problems, an information processing device according to an aspect of the present invention includes: an image acquisition unit that captures images of a crane transporting garbage after grabbing the garbage with a grab bucket and rolling up the garbage in the state of grabbing the garbage. An image of a series of states up to the grab bucket; and a detection unit that detects that the grab bucket can be rolled up based on an output value obtained by inputting the above image to a state judgment model, wherein the state judgment model is Constructed by machine learning using an image of a grab bag in a state where garbage is picked up and rolled up as teacher data.

And, in order to solve the above-mentioned technical problems, the information processing method of one aspect of the present invention uses an information processing device to perform information processing, and includes: an image acquisition step, which captures and captures images of a crane carrying garbage after grabbing it with a grab bucket. An image of a series of states until the grab bucket is rolled up in the state of garbage; and a detection step, which detects the situation that the grab bucket can be rolled up based on the output value obtained by inputting the above image to the state judgment model, wherein The state judgment model is constructed by machine learning using an image of a grab bucket in a state where garbage has been picked up and rolled up as teacher data.

(3) Beneficial effects

According to one aspect of the present invention, it is possible to reduce the time loss from the start of rolling up until the rolling is performed, compared with the case of rolling up by control using a timer or the like.

Description of drawings

FIG. 1 is a diagram showing an overview of a control system according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an example of a configuration of main parts of an information processing device included in the control system.

FIG. 3 is a block diagram showing an example of a configuration of main parts of a control device included in the control system.

FIG. 4 is a block diagram showing an example of a configuration of main parts of a learning device included in the control system.

Fig. 5 is a diagram showing a control history of rolling up of the grab bucket by manual control.

FIG. 6 is a diagram showing an example of an image usable as teacher data for constructing a state determination model.

7 is a diagram showing an example of judging the state of the grab bucket (horizontal/inclined, open/half open, mid-air) using a machine-learning model.

Fig. 8 is a diagram showing an example of judging the state of the grab (horizontal, open/closed, airborne/grounded) using a machine-learning model.

FIG. 9 is a flowchart showing an example of processing of the learning device constructing a state determination model.

FIG. 10 is a flowchart showing an example of processing executed by the information processing device.

Fig. 11 is a flowchart showing an example of processing executed when the garbage at the garbage picking position is dropped to the dropping position by a crane under automatic control of the control device.

Fig. 12 is a diagram showing the transition of the control history for the crane and the result of state determination by the above-mentioned information processing device.

Detailed ways

(System Outline)

An overview of a control system 100 according to one embodiment of the present invention will be described based on FIG. 1 . FIG. 1 is a diagram showing an overview of a control system 100 . In addition, FIG. 1 also shows an outline of a waste incineration facility including a crane 93 for conveying waste, and the crane 93 is a control object of the control system 100 .

As shown in FIG. 1 , the garbage incineration facility includes a garbage pit 91 for temporarily storing garbage brought in by a collection vehicle P, and a garbage incinerator 92 . The garbage pit 91 is connected with the incinerator 92 by a hopper, and the garbage in the garbage pit 91 is sent into the incinerator 92 for burning through the hopper.

The bottom of the garbage pit 91 serves as a storage section for garbage, and the collection vehicle P pours garbage into the storage section from the carrying door, and the garbage is stored in the storage section (garbage G in the figure). Moreover, the crane 93 is installed in the zenith part of the garbage pit 91. As shown in FIG. The crane 93 includes a main girder 931 , a traverse crane 932 , a wire rope 933 , and a grab bucket 934 .

Guide rails (extending in the depth direction of the drawing) are respectively provided on opposing wall surfaces of the building of the garbage pit 91, and the main beam 931 is arranged so as to span between the guide rails. The main beam 931 is movable along the guide rail in the depth direction of the figure. In addition, the traversing crane 932 is arranged on the main beam 931 . The traverse crane 932 can move on the main girder 931 in the left-right direction (direction perpendicular to the moving direction of the main girder 931 ) in the figure. A hoist (for example, a winch) is placed on the traversing crown 932 . A grab bucket 934 for grabbing garbage G is provided at the tip of a wire rope 933 extending from the hoist. The grab 934 is capable of opening and closing.

In this way, the main beam 931 can move in the depth direction of the figure, and the traverse crane 932 can move in the left and right directions of the figure, so the combination of these movements can be used to move the grab bucket 934 to any position in the storage part. In addition, the wire rope 933 is extended from the hoist, the grab bucket 934 is lowered, and the garbage G in the storage part can be grabbed by the grab bucket 934 . And, by controlling the operation of the main beam 931, the traverse crane 932, the grab bucket 934, and the hoist, the grabbed garbage G can be transferred to another position in the storage part or thrown into a hopper.

The operation control of the crane 93 in this way can be performed from the operation room 94 provided in the side wall part of a building so that the inside of a storage part can be monitored. A control system 100 is installed in the operator room 94 . The control system 100 includes an information processing device 1 , a control device 3 , a learning device 5 , and a photographing device 7 .

When controlled by the control system 100 , the photographing device 7 photographs a series of states from when the crane 93 carrying garbage grabs the garbage with the grab bucket 934 to when the garbage grab 934 is rolled up. For example, the imaging device 7 may capture moving images of the grab bucket 934 and its surroundings during the period from picking up garbage to rolling up the grab bucket 934 in a state of picking up garbage, so as to capture the above-mentioned "series of states". In addition, the photographing device 7 may photograph the still images of the grab bucket 934 and its surroundings at predetermined intervals during the above-mentioned period, thereby photographing the above-mentioned "series of states".

In addition, the said "a series of states" includes the state which cannot be rolled up, and the state which can be rolled up. The state that cannot be rolled up is the state in which the garbage grabbed by the grab bucket 934 cannot be separated from other garbage piled up around but connected. Since it cannot be rolled up in this state, the garbage picked up by the grab bucket 934 is separated from other garbage by performing a garbage picking operation which will be described later, so that it can be rolled up. The above-mentioned "series of states" may also include a state of performing garbage capture.

It can also be said that the above-mentioned period "until the grab bucket is rolled up" is the period until the moment when the grab bucket 934 leaves the ground. Lifting off the ground refers to the state in which the grabbed garbage is separated from the garbage layer and can roll up the grab bucket 934 . Of course, the photographing device 7 only needs to perform photographing at least until the garbage is picked up by the grab bucket 934 until the garbage is lifted off the ground.

The information processing device 1 detects that the grab bucket 934 can be rolled up by inputting the output value obtained by the captured image into a state judgment model constructed by machine learning. The above-mentioned machine learning is performed by the learning device 5 using an image of a grabbing bucket that has picked up garbage and can be rolled up as teacher data, which will be described in detail later.

Moreover, the control device 3 causes the crane 93 to perform the winding-up operation of the grab bucket 934 when the information processing device 1 detects that the grab bucket 934 can be rolled up.

In addition, the rolling-up operation|movement of the grab bucket 934 means the operation|movement which rolls up the grab bucket 934 to the height which can move in a horizontal direction. In order to be able to roll up the grab bucket 934 after grabbing garbage with the grab bucket 934, sometimes the grab bucket 934 is rolled up at a low speed to the extent that the grab bucket 934 is slightly suspended in the air, but this kind of rolling is not an information processing device. 1 detection object.

According to the above-mentioned configuration, it is automatically detected that the grab bucket 934 is in a rollable state from the image of a series of states after the grab bucket 934 grabs the garbage to rolling up the grab bucket 934 that has grabbed the garbage. and roll up. Therefore, compared with the case where rolling is carried out under control using a timer or the like, it is possible to reduce the time loss from the start of rolling up until the rolling is performed.

1 shows an example in which the information processing device 1, the control device 3, the learning device 5, and the photographing device 7 are arranged in the operator's room 94. In addition to the control device 3 that is also used for the manual control of the crane 93, this The configuration is not particularly limited. For example, the information processing device 1 and the learning device 5 may be arranged in an external remote monitoring facility of a waste incineration facility or the like.

(Structure of Information Processing Device)

The configuration of the information processing device 1 will be described based on FIG. 2 . FIG. 2 is a block diagram showing an example of the configuration of main parts of the information processing device 1 . As shown in the figure, the information processing device 1 includes: a control unit 10 that comprehensively controls each unit of the information processing device 1; a storage unit 11 that stores various information used by the information processing device 1; an input unit 12 that accepts instruction input; and a communication unit 13, which is used for wireless communication with other devices. In addition, the control unit 10 includes: an image acquisition unit 101, an inclination state determination unit 102, an inclination detection unit 103, a crane state determination unit 104, a ground clearance detection unit (detection unit) 105, an open/closed state determination unit 106, and The detection unit 107 is completely held.

The image acquiring unit 101 acquires an image captured by the photographing device 7 , that is, an image of a series of states from when the crane 93 grabs the garbage with the grab bucket 934 to when the grab bucket 934 is rolled up in a state of grabbing garbage. In addition, when the imaging device 7 captures a moving image, the image acquisition unit 101 may acquire frame images from the moving image at predetermined time intervals.

The inclination state determination unit 102 determines the inclination state of the grab bucket 934 captured in the image based on the image acquired by the image acquisition unit 101 . For example, the inclination state determination unit 102 may determine the inclination state using an inclination determination model constructed by machine learning. Construction of the inclination determination model will be described later based on FIG. 7 and the like.

The inclination detection part 103 detects that the grab bucket 934 which exists on the refuse surface inclines based on the determination result of the inclination state determination part 102. Then, the inclination detection unit 103 notifies the control device 3 of the detection result.

Thus, since the information processing device 1 includes the inclination state determination unit 102 and the inclination detection unit 103 , the inclination of the grab bucket 934 on the surface of the garbage can be detected based on the image of the grab bucket 934 captured by the imaging device 7 . And the control apparatus 3 performs the control which changes the garbage pick-up position of the crane 93, when the information processing apparatus 1 detects that the grab bucket 934 inclines. According to this structure, when the inclination of the grab bucket 934 is detected, the grabbing position of the garbage is changed, so it is possible to prevent the overturning of the grab bucket 934 and the grabbing of garbage caused by the garbage grabbing operation in the tilted state. loss etc.

The crane state determination unit 104 determines the state of the grab bucket 934 captured in the image based on the image acquired by the image acquisition unit 101 . Specifically, the crane state determination unit 104 calculates an output value indicating the state of the grab bucket 934 by inputting the image acquired by the image acquisition unit 101 into the state determination model 514 constructed by the learning device 5 . Furthermore, the state determination model 514 is constructed by machine learning using the image of the grab bucket 934 in the state of being able to pick up and roll up garbage as the teacher data 513 , which will be described in detail later.

The ground lift detection unit 105 detects that the grab bucket 934 is in a state where it is possible to wind up the grab bucket 934 based on the above output value from the state determination model 514 . Then, the ground lift detection unit 105 notifies the control device 3 of the detection result. In addition, the state in which the grab bucket 934 which has picked up garbage and can be rolled up is also called a ground-off state.

In this way, the information processing device 1 includes the image acquisition unit 101 that captures images of a series of states from when the crane 93 grabs the garbage with the grab bucket 934 to when the grab bucket 934 is rolled up in the garbage-grabbed state. image. Moreover, the information processing apparatus 1 is equipped with the ground lift detection part 105 which detects that the grab bucket 934 has become the state which can roll up the grab bucket 934 based on the output value which input the said image to the state determination model 514. The state determination model 514 described above is constructed by machine learning using an image of the grab bucket 934 in a state where garbage is picked up and can be rolled up as teacher data.

According to the above-mentioned structure, it is automatically detected that the roll-up grab 934 is in a roll-up state from an image of a series of states after the grab bucket 934 grabs the garbage to rolling up the grab bucket 934 that has picked up the garbage. Case. Therefore, the control device 3 can be turned up by this detection as a trigger, thereby reducing the time loss from the time when the winding is possible until the winding is performed, compared with the case where the winding is performed under the control of a timer.

The opening and closing state determination unit 106 determines the opening and closing state of the grab bucket 934 captured in the image based on the image acquired by the image acquisition unit 101 . For example, the open/close state determination unit 106 may determine the open/close state using an open/close determination model constructed by machine learning. Construction of the opening/closing determination model will be described later based on FIG. 7 and the like.

The incomplete holding detection unit 107 detects that the degree of closure of the grab bucket 934 is insufficient when picking up garbage based on the determination result of the opening/closing state determination unit 106 . Then, the incomplete holding detection unit 107 notifies the control device 3 of the detection result.

In this way, information processing device 1 includes open/closed state determination unit 106 and incomplete retention detection unit 107 , and therefore can detect that grab bucket 934 is closed enough to hold garbage based on the image acquired by image acquisition unit 101 . And, when the information processing device 1 detects that the grab bucket 934 is closed enough to hold the garbage, the control device 3 performs control to close the grab bucket 934 .

Here, the opening degree of the grab bucket 934 can be measured by the hydraulic pressure when opening and closing the grab bucket 934, and the opening degree of the grab bucket 934 can be adjusted according to the measured value. However, the grab 934 may be in an open state with respect to the measured value due to the garbage being held by the claws of the grab 934 or the like. Therefore, according to the above-mentioned structure, when it detects that the degree of closure of the grab bucket 934 is insufficient to hold garbage, the control to close the grab bucket 934 is performed. Thereby, it is possible to prevent the loss of picking up the garbage or the dropping of the garbage to an undesired position.

(Structure of the control unit)

The configuration of the control device 3 will be described based on FIG. 3 . FIG. 3 is a block diagram showing an example of the configuration of main parts of the control device 3 . As shown in the figure, the control device 3 includes a control unit 30 , a storage unit 31 , an input unit 32 , an output unit 33 , and a communication unit 34 . The control unit 30 comprehensively controls each unit of the control device 3 . The storage unit 31 stores various information used by the control device 3 . The input unit 32 accepts user's instruction input. The output unit 33 is used for the control device 3 to output information. The communication unit 34 is used for wireless communication with other devices. In addition, the control unit 30 includes a crane control unit 301 , a grab bucket movement control unit 302 , a grab bucket opening and closing control unit 303 , and a grab bucket lifting control unit 304 .

The crane control unit 301 controls the operation of the crane 93 . Specifically, the crane control unit 301 specifies the picking position of the garbage picked up by the crane 93 and the dropping position of the picked up garbage. And, the crane control unit 301 controls the grab bucket movement control unit 302, the grab bucket opening and closing control unit 303, and the grab bucket lifting control unit 304, so that the crane 93 grabs the garbage at the grab position and The above-mentioned dropping position throws garbage. There is no particular limitation on the method of determining the picking position and dropping position of the garbage. For example, it can be determined based on the stirring state and accumulation height of the garbage at each position in the garbage pit, or it can be determined according to a preset schedule.

The grab movement control unit 302 moves the main girder 931 and the traverse crane 932 according to the control of the crane control unit 301 , and moves the grab bucket 934 to the position directly above the garbage grabbing position or the dropping position.

The grab opening and closing control unit 303 changes the opening degree of the grab bucket 934 according to the control of the crane control unit 301 . In this embodiment, the grab opening and closing control unit 303 makes the grab 934 open by sending a positive opening and closing command to the crane 93, and makes the grab 934 a closed state by sending a negative opening and closing command. . Moreover, the opening and closing state of the grab bucket 934 is maintained in the current state by setting the value of the opening and closing command to the crane 93 as a neutral position (0). Of course, these controls are merely examples, and any control may be performed in order to open and close the grab bucket 934 .

The grab lifting control unit 304 raises and lowers the grab bucket 934 according to the control of the crane control unit 301 . Moreover, when raising and lowering the grab bucket 934, the grab bucket raising and lowering control part 304 raises and lowers the grab bucket 934 at the raising and lowering speed specified by the crane control part 301.

(Structure of learning device)

The configuration of the learning device 5 will be described based on FIG. 4 . FIG. 4 is a block diagram showing an example of the configuration of main parts of the learning device 5 . As shown in the figure, the learning device 5 is equipped with: a control unit 50 that comprehensively controls each part of the learning device 5; a storage unit 51 that stores various information used by the learning device 5; an input unit 52 that accepts user's instruction input; and The output unit 53 is used for the learning device 5 to output information. In addition, the control unit 50 includes a history extraction unit 501 , a time determination unit 502 , an image extraction unit 503 , a teacher data generation unit 504 , and a learning unit 505 . Furthermore, an operation history DB (database) 511 and a pit image 512 are stored in the storage unit 51 . Furthermore, teacher data 513 and a state judgment model 514 are stored.

The operation history DB511 is a database in which the control history of the crane 93 is recorded. More specifically, control history data showing the content of the control performed on the crane 93 and the time when the control was performed is recorded in the operation history DB 511 . In addition, whether each control is performed manually or automatically is also recorded in the operation history DB 511 .

The pit image 512 is an image of the inside of the garbage pit captured by the photographing device 7 during the operation of the refuse incineration facility. The pit image 512 includes images showing a series of states from when the crane 93 grabs garbage with the grab bucket 934 to when the garbage grab 934 is picked up is rolled up.

The teacher data 513 is teacher data used in machine learning for constructing the state determination model 514 . The state determination model 514 only needs to be able to determine at least the ground-off state, so the teacher data 513 only needs to be data capable of at least learning the ground-off state. In addition, the details of the teacher data 513 will be described later based on FIG. 6 .

The state judgment model 514 is a model used when judging whether or not the grab bucket 934 is in a state (off-the-ground state) that grabs garbage and can be rolled up. The details of the state determination model 514 will also be described later together with the details of the teacher data 513 based on FIG. 6 .

The history extraction part 501 extracts the control history data of the manual winding control from operation history DB511. As described above, whether each control is manually performed or automatically performed is recorded in the operation history DB 511 , and therefore the history extraction unit 501 can extract control history data of manually performed winding control based on this record.

The time specifying unit 502 specifies the time when the winding control was manually performed based on the control history data extracted by the history extracting unit 501 . A method of specifying the timing at which the winding control is manually performed will be described later based on FIG. 5 .

The image extracting section 503 extracts an image captured at the timing specified by the timing specifying section 502 from the pit image 512 . For example, when the pit image 512 is composed of a sequential still image group captured continuously, the image extracting unit 503 extracts an image captured at a time specified by the time specifying unit 502 from the still image group. On the other hand, when the pit image 512 is a moving image, the image extracting unit 503 extracts the frame image at the time specified by the time specifying unit 502 from the moving image.

The teacher data generating unit 504 associates labels of correct data with the images extracted by the image extracting unit 503 to create teacher data 513 used when constructing the state determination model 514 . A specific method of generating teacher data 513 will be described later based on FIG. 6 .

The learning unit 505 uses the teacher data 513 generated by the teacher data generating unit 504 to perform machine learning and construct a state determination model 514 . In addition, the method of machine learning and the type of model to be constructed are not particularly limited as long as object detection and object state determination can be performed from images. For example, the state decision model 514 can be constructed using a convolutional neural network. The same is true for the inclination determination model and the opening and closing determination model.

As described above, the learning device 5 includes the time specifying unit 502 that refers to the control history data of the crane 93 to specify the time when the operator manually operates the crane 93 to pick up and roll up garbage with the grab bucket 934 . In addition, the learning device 5 includes a teacher data generation unit 504 . The teacher data generating unit 504 uses the image of the grab 934 taken at the time specified by the time specifying unit 502 as the teacher data 513 for constructing the state determination model 514 for detecting the rollable state of the grab 934 use.

According to the above-mentioned structure, the image of the grab bucket 934 image|photographed at the time when an operator manually operated the crane 93 to pick up garbage and rolled it up is set as the teacher data 513. FIG. Therefore, by performing machine learning using the teacher data 513 and constructing the state determination model 514 , it is possible to detect the state in which the grab bucket 934 can be rolled up at the same timing as when the operator performs an operation.

(the determination method of the roll-up time)

A method for determining the winding time by the time determination unit 502 will be described based on FIG. 5 . FIG. 5 is a diagram showing a control history of rolling up of the grab bucket 934 by manual control. The horizontal axis of FIG. 5 is the time, and the vertical axis is the roll-up position of the grab bucket 934 . The roll-up gears are four stages of 0, 1, 2, and 3. Roll-up is not performed at 0, and roll-up is performed at 1-3. The larger the value of the roll-up gear, the faster the roll-up speed.

In the control history of FIG. 5 , at 14:06:50, the winding position was set to 1, and this state continued until 14:06:53. That is, low-speed winding is performed during this period. And, from 14:06:53 to 14:06:54, the roll-up gear is switched from 1 to 3, and then the roll-up gear maintains the state of 3. That is, high-speed winding is performed during this period.

Here, generally, when the operator of the crane picks up the garbage, the roll-up position is set at a low speed to lift off the ground, and when it is judged that the lift-off is completed, the roll-up position is set to a high-speed roll. Grab 934.

Based on this, it can be considered that in the manual control of the control history shown in Figure 5, the lift-off is performed during the period when the roll-up gear is 1, and at the moment when the roll-up gear is switched from 1 to 3 at 14:06:54 (point The moment of P1) starts to roll up. Therefore, the timing determination unit 502 can determine the timing when the winding position is shifted to 3, that is, the timing when the manual control is performed at a predetermined speed normally applied during winding, as the winding timing.

Therefore, what is necessary is just to include in the operation history DB511 the data which can specify the time when the manual control which was set to predetermined winding speed was performed. This data may be data as shown in FIG. 5 , or may be data in which the time when the manual control was performed and the content of the manual control are associated.

(for images used for machine learning)

The time determined by the timing specifying unit 502 as described above is the time immediately after the liftoff is completed, and therefore the image of the grab bucket 934 taken at this time is an image showing the state where the liftoff has been completed, that is, the liftoff state. Therefore, the image extracting unit 503 extracts, from the pit image 512 , an image captured at the time specified by the time specifying unit 502 as an image of the ground lift state.

For example, when the time specifying unit 502 determines that the time is 14:06:54 based on the control history in FIG. 5 , the image extracting unit 503 extracts an image captured at that time as an image of the ground-off state.

Furthermore, the teacher data generating unit 504 uses the image captured at the time determined by the time determining unit 502 and extracted by the image extracting unit 503 as described above as the teacher data 513. The model 514 is used to detect the rollable state of the grab bucket 934 . Specifically, the teacher data generation unit 504 creates teacher data 513 by associating a label indicating the ground-off state with the image extracted by the image extraction unit 503 as the image of the ground-off state as correct data.

In addition, the image captured immediately before the time determined by the time specifying unit 502 is an image showing a grounded state in which the grab bucket 934 is in contact with the surface of the garbage layer accumulated in the garbage pit. Therefore, the image extracting unit 503 can extract, from the pit image 512 , an image captured during a predetermined period immediately before the time determined by the time specifying unit 502 as an image showing the grounded state. In addition, the start of the predetermined period can be set, for example, at the time when it is determined that the grab 934 is grounded. Grounding of the grab bucket 934 can be determined based on, for example, an output value of a weight sensor provided at a connection portion between the wire rope 933 and the grab bucket 934 of the suspension grab bucket 934 .

For example, when the time of 14:06:54 is determined by the time determination unit 502 based on the control history in FIG. image), as an image of the grounded state. In this case, the teacher data generating unit 504 creates teacher data 513 by associating a label indicating the grounded state as correct data with the image extracted by the image extracting unit 503 as an image of the grounded state.

FIG. 6 is a diagram showing an example of an image that can be used as teacher data 513 for constructing a state determination model 514 . More specifically, among the images shown in FIG. 6 , IMG1 is an example of an image for learning a grounded state, and IMG2 is an example of an image for learning a grounded state.

The shooting time of each image is recorded on the upper left of IMG 1 and 2 . From this moment, IMG1 was shot at 14:06:53 on October 9, 2018, and IMG2 was shot at 14:06:54 on the same day.

In addition, in IMG1 and 2, areas A1 and A2 are set so as to surround the grab bucket 934 . In the case where the state determination model 514 is set as a model that detects the grab bucket 934 from an image in the garbage pit and determines the state of the detected grab bucket 934, information indicating such areas A1 and A2 can be used as The correct data only needs to be included in the teacher data 513 . In this case, the information indicating the areas A1 and A2 may be generated by image analysis or the like, or may be input by the user of the learning device 5 through the input unit 52 .

In addition, since IMG1 is an image of the grab 934 in the grounded state, it is possible to create the teacher data 513 in association with a tag indicating that it is in the grounded state. In addition, since IMG2 is an image of the grab 934 in the off-the-ground state, the teacher data 513 can be created in association with a tag indicating the off-the-ground state.

Similarly, teacher data can be generated from images of various states of the grab 934 , and various states of the grab 934 can be determined by performing machine learning using the teacher data. 7 and 8 are diagrams showing an example of determining the state of the grab bucket 934 using a machine-learning model.

In the image of IMG3 shown in FIG. 7 , area A3 in which grab bucket 934 is captured is detected, and it is determined that grab bucket 934 is tilted horizontally and the claws for grabbing garbage are open and in the air (not grounded).

In addition, in the image of IMG4, the area A4 in which the grab 934 is photographed is detected, and it is determined that the grab 934 is not horizontal but inclined, and the claws for picking up garbage are in a half-open state.

In addition, in the image of IMG5 shown in FIG. 8 , the area A5 in which the grab 934 is captured is detected, and it is determined that the inclination of the grab 934 is horizontal, the claws for grabbing garbage are closed, and the grab 934 is grounded.

In addition, in the image of IMG6, the area A6 in which the grab 934 is imaged is detected, and it is determined that the grab 934 is horizontal, the claws for grabbing garbage are opened, and the state is in the air (not grounded).

The teacher data 513 can be created by associating with a tag indicating that the grab 934 is in the air like IMG3 in FIG. 7 and IMG6 in FIG. 8 . Which of the three states the grab bucket 934 is in can be determined by using the above-mentioned teacher data 513 . That is to say, the above-mentioned teacher data 513 is: (1) the teacher data 513 associated with the label representing the off-the-ground state; (2) the teacher data 513 associated with the label representing the grounded state; Teacher data 513 of the label of the air state.

In addition, the state determination model 514 should just make it possible for the ground lift detection part 105 to detect that it becomes the state which can wind up the grab bucket 934 from the output value of the state determination model 514. For example, the state decision model 514 may only output the probability of being off the ground. In this case, what is necessary is just to make the ground lift detection part 105 detect that the grab bucket 934 can be rolled up, when the probability that it is a lift state becomes more than a threshold value.

In addition, for example, the state decision model 514 may output the probability that the grab 934 is in the grounded state and the probability that it is in the air state. In this case, as long as the ground separation detection unit 105 detects that the grab bucket 934 can be rolled up when the probability of being in the grounded state and the probability of being in the air state are both less than the threshold value, that is, it is difficult to say that it is in the grounded state or in the air state. The status of the situation can be.

Also, for example, the state determination model 514 may output only the probability that the grab bucket 934 is in the grounded state. In this case, it is only necessary to enable the ground lift detection unit 105 to detect when the probability of being a grounded state changes from a state in which the probability of being a grounded state is greater than or equal to a threshold value to a state in which the probability of being a grounded state is less than a threshold value, that is, when it is difficult to say that the grounded state has changed from a grounded state to a grounded state. The case where the grab bucket 934 is rolled up is sufficient.

Moreover, with respect to the image which captured the grab 934 in a horizontal state like IMG3 of FIG. 7, and IMG5, 6 of FIG. Moreover, with respect to the image which captured the grab bucket 934 in a tilted state like IMG4 of FIG.

And, by performing machine learning using such teacher data, it is possible to construct a tilt determination model. By inputting the image of the grab 934 captured in the inclination determination model constructed in this way, the probability that the grab 934 captured in the image is in an inclined state and the probability that it is in a horizontal state can be output.

In addition, as long as the inclination determination model enables the inclination detection unit 103 to detect the inclination of the grab bucket 934 to such an extent that it is highly likely that the grab bucket 934 may not be able to grasp garbage based on the output value of the inclination determination model, for example, it cannot be achieved by a skilled operator. It is only necessary to perform the rubbish picking operation.

For example, the inclination determination model may be constructed using, as teacher data, an image of the grab bucket 934 tilted to such an extent that it is considered that there is a high possibility that the grab bucket 934 cannot catch garbage. In this case, the inclination detection part 103 should just detect that the grab bucket 934 inclines, when the probability value output from the inclination determination model that it is an inclination state is more than a threshold value.

In addition, for example, the inclination determination model may be constructed using an image of the grab bucket 934 in a horizontal state as teacher data. In addition, in this case, the image of the grab bucket 934 tilted to such a degree that it is considered that the possibility of not being able to grasp the garbage is low, in other words, to the extent that a skilled operator can perform the garbage picking operation, can also be regarded as level state for learning. When such learning is performed, the inclination detection unit 103 may detect that the grab bucket 934 is tilted when the probability value of the horizontal state output from the inclination determination model is less than a threshold value, that is, when the state is hardly horizontal.

Moreover, with respect to the image which captured the grab 934 in the open state like IMG3 of FIG. 7 and IMG6 of FIG. 8, the tag which shows that it is an open state can be associated and can be created as teacher data. Moreover, with respect to the image which captured the grab bucket 934 in a closed state like IMG5 of FIG. Similarly, for an image of the grab 934 in a half-open state like IMG4 in FIG. 7 , it is possible to create teacher data in association with a tag indicating that it is in a half-open state.

Furthermore, by performing machine learning using such teacher data, it is possible to construct an open/close determination model. By inputting an image captured by the grab 934 into the opening/closing determination model constructed in this way, the probability that the grab 934 captured in the image is in an open state, the probability that it is in a closed state, and the probability that it is in a half-open state can be output. .

In addition, the opening/closing determination model should just make it possible for the incomplete holding|maintenance detection part 107 to detect that the grab bucket 934 has the degree of closure which is insufficient to hold waste from the output value of this opening/closing determination model. For example, an open/close decision model may only output the probability of being a half-open state. In this case, the incomplete holding detection unit 107 may detect that the degree of closure is insufficient when the probability of being in the half-open state is equal to or greater than a threshold value.

In addition, for example, the open/close judgment model may output the probability that the grab bucket 934 is in the open state and the probability that it is in the closed state. In this case, the incomplete hold detection unit 107 may detect that the degree of closure is insufficient when both the probability of being in the open state and the probability of being in the closed state are less than a threshold value, that is, when it is difficult to say whether it is in the open state or the closed state. .

In addition, the state judgment model 514, the inclination judgment model, and the opening/closing judgment model may be constructed as independently learned models. Alternatively, these models can be combined as one learned model. For example, these three models can be combined as one learned model. In this case, the area where the grab 934 is photographed can be detected from the image in which the grab 934 is photographed, and for this grab 934, the model output can be made to correspond to tilt/level, open/close/half Probability values for each state of open, air/grounded/leaved. When such a model is used, the inclination state determination unit 102, the crane state determination unit 104, and the switch-closed state determination unit 106 in the block diagram of FIG. 2 are integrated into one module.

(Flow of processing (construction of state judgment model))

The flow of processing (learning method) in which the learning device 5 constructs the state determination model 514 will be described based on FIG. 9 . FIG. 9 is a flowchart showing an example of the process of constructing the state determination model 514 by the learning device 5 . In addition, it is set that control history data of the crane 93 for a predetermined period is recorded in the operation history DB 511 before the process of FIG. .

In S1, the history extraction part 501 extracts the control history data of the winding control performed manually from the control history data recorded in the operation history DB511. Next, in S2, the time specification part 502 specifies the time when this winding control was performed with respect to the control history data extracted in S1.

In S3 , the image extracting unit 503 extracts the image captured at the time specified in S2 from the pit image 512 . In this image, like the example of IMG2 in FIG. 6 , the grab bucket 934 in a state of being lifted off the ground is photographed.

In S4, the teacher data generator 504 associates with the image extracted in S3 as correct data a label indicating that it is off the ground to make teacher data 513. This teacher data 513 is used when constructing the state judgment model 514. The state judgment A model 514 is a model for detecting a rollable state.

In S5 , the learning unit 505 performs machine learning using the teacher data 513 generated in S4 , and builds a state determination model 514 . Then, in S6, the learning unit 505 records the state determination model 514 constructed in S5 in the storage unit 51, thereby ending the processing of FIG. 9 .

Moreover, in S2, if it is the structure which identified the time immediately before the time when the wind-up control was performed, the image which captured the grab bucket 934 in the grounded state is extracted in S3. In this case, in S4, the teacher data generator 504 associates the label indicating the grounded state with the image extracted in S3 as correct data to create teacher data 513, which is used when the state judgment model 514 is constructed. use.

Also, in S2, if the grab bucket 934 is configured to determine a certain timing (for example, the timing between when the grab bucket 934 is controlled to roll up and when the grab bucket 934 is lowered) is configured, then in S3 Extract the image of the grab bucket 934 in the aerial state. In this case, in S4, the teacher data generating unit 504 associates a label indicating the air state with the image extracted in S3 as correct data, and creates this teacher data 513 for use in constructing the state judgment model 514 .

In addition, in the above-mentioned flowchart, the control history data of the manually performed winding control is extracted in S1, but if the winding control is performed at an appropriate timing, the control history data of the automatically performed winding control may be extracted. That is, the history extracting part 501 should just extract the control history data of the winding control performed at an appropriate timing from the control history data recorded in the operation history DB511 in S1.

(Processing flow (status judgment and notification))

The flow of processing (information processing method) executed by the information processing device 1 will be described based on FIG. 10 . FIG. 10 is a flowchart showing an example of processing executed by the information processing device 1 . In addition, during the operation period of the crane 93, the control device 3 executes the processing of FIG. 11 described later in parallel with the information processing device 1 executing the processing of FIG. 10 .

In S11 (image acquisition step), the image acquisition unit 101 acquires an image of the crane 93 captured by the imaging device 7 . The acquired image is a still image or a frame image extracted from a moving image. In addition, the processes of S11 to S16 are repeated at least during the automatic control period of the crane 93 . Therefore, the process of S11 is also repeated to acquire images of a series of states from when the crane 93 grabs the garbage with the grab bucket 934 to when it rolls up the grab bucket 934 in a state of grabbing garbage.

In S12, the inclination state determination part 102 inputs the image acquired in S11 to the inclination determination model, and determines the inclination state of the grab bucket 934. FIG. And the inclination detection part 103 detects that the grab bucket 934 is in an inclined state in the control apparatus 3, when the output value of an inclination determination model shows that the possibility that the grab bucket 934 is in an inclined state is more than a threshold value.

In S13 (detection step), the crane state determination unit 104 inputs the image acquired in S11 to the state determination model 514 to determine the state of the crane 93 . And the ground lift detector 105 detects that the grab bucket 934 can be rolled up, when the output value of the state determination model 514 shows that the possibility of being a lift state is more than a threshold value.

In S14, the opening and closing state determination part 106 inputs the image acquired in S11 to the opening and closing state determination model, and determines the opening and closing state of the grab bucket 934. FIG. In addition, the incompletely held detection unit 107 detects the incompletely held state when the output value of the open/close state determination model shows that the possibility that the grab bucket 934 is in the half-open state is equal to or greater than a threshold value. In addition, the execution order of the process of S12-S14 is arbitrary. In addition, a configuration in which these processes are performed in parallel may also be employed.

In S15 , the control device 3 is notified of the contents of the determination results in S12 to S14 that need to be notified to the control device 3 . Specifically, when detecting that the grab bucket 934 is in an inclined state in S12, the inclination detection part 103 notifies the controller 3 of this fact. Moreover, when the ground lift detection part 105 detects that it is the state which can roll up the grab bucket 934 in S13, it notifies the control apparatus 3 of this fact. In addition, when the incomplete holding detection part 107 detects that it is an incomplete holding state in S14, it notifies the control apparatus 3 of that.

In S16, the image acquisition part 101 determines whether the automatic control of the crane 93 is complete|finished. The criterion for determining whether to end the automatic control is not particularly limited. For example, it may be determined that the automatic control is ended when an operation is performed on the input unit 12 to instruct the end of the automatic control. In addition, since the determination results of S12 to S14 can be used as a reference at the time of manual control, the process of FIG. 10 can be continued while the crane 93 is in operation. In this case, in S16, it is only necessary to determine whether or not to end the operation of the crane 93 .

When it is judged to end in S16 (YES in S16), the process of FIG. 10 ends. On the other hand, when it is determined not to end (No in S16), the process returns to S11, and the image acquisition unit 101 acquires a new image (an image next to the image acquired in the previous S11 in terms of chronological order). ).

As described above, the information processing method of the information processing device 1 includes an image acquisition step (S11) in which the crane 93 arrives after grabbing the garbage with the grab bucket 934 and grabs the garbage. A series of images of states until the grab bucket 934 is rolled up. In addition, this information processing method includes a detection step (S13) in which the state in which the grab bucket 934 can be rolled up is detected based on the output value obtained by inputting the above-mentioned image to the state judgment model 514. .

Therefore, the control device 3 can be caused to wind up by the notification of the detection result in S15 based on the detection result in S13. Thereby, compared with the case where rolling is performed by control using a timer etc., time loss from the start of being able to roll up to rolling up can be reduced.

(Flow of processing (crane control))

The flow of processing (control method) executed when the control device 3 automatically controls the operation of the crane 93 will be described based on FIG. 11 . FIG. 11 is a flowchart showing an example of processing executed when the garbage at the garbage picking position is dropped to the dropping position by the crane 93 under the automatic control of the control device 3 .

In S21, the crane control part 301 specifies a garbage pick-up position. And the crane control part 301 instructs the grab bucket movement control part 302 to move the grab bucket 934 to the specified said position. Thereafter, the crane control unit 301 instructs the grab opening and closing control unit 303 to bring the grab bucket 934 into an open state, and instructs the grab bucket raising and lowering control unit 304 to lower the grab bucket 934 . Thereby, the grab 934 will be in the state which contact|connected with the refuse surface.

Here, while the crane 93 is operating in accordance with the control in S21, the photographing device 7 photographs the situation. Then, the image captured by the photographing device 7 is sent to the information processing device 1, and the information processing device 1 performs the processing of FIG. 10 using the image, and notifies the control device 3 of the determination result as needed. Based on such a notification, the following processing of the control device 3 is performed.

In S22, the crane control unit 301 determines whether or not to change the garbage picking position. Specifically, when the crane control unit 301 receives the detection result that the grab bucket 934 is in an inclined state from the information processing device 1, it determines to change the garbage picking position (YES in S22), and the process proceeds to S23. On the other hand, when the crane control part 301 does not receive such a detection result, it determines with not changing a garbage pick-up position (No in S22), and a process progresses to S24.

In S23, the crane control part 301 changes a garbage pick-up position, and a process returns to S21. The method of changing the garbage collection position is not particularly limited. For example, the crane control unit 301 may change the garbage picking position to a position shifted by a predetermined distance from the previous garbage picking position.

In S24, the crane control unit 301 causes the grab opening and closing control unit 303 to start the garbage picking operation. The content of the garbage collection operation may be determined in advance. For example, the garbage picking operation may be an operation realized by performing an opening command and a closing command to the grab bucket 934 a predetermined number of times in a predetermined time.

In S25, the crane control part 301 determines whether to carry out the winding up of the grab bucket 934 or not. Here, when the crane control unit 301 detects that the grab bucket 934 is in a rollable state after a predetermined lower limit period has elapsed after the garbage picking operation starts and the information processing device 1 detects (in S25 for yes). In addition, the predetermined lower limit period may be appropriately determined with reference to, for example, the time required for general ground lift. That is, the predetermined lower limit period may be such a period that it is difficult to think that the vehicle can be lifted from the ground in consideration of the time generally required to lift off the ground.

In S25, even if the information processing device 1 detects that the grab bucket 934 can be rolled up and notifies the control device 3 of this fact, if the notification time is before the predetermined lower limit time has elapsed, no judgment is made. Roll up for execution. Thereby, for example, even if a rollable state is detected although it is not a rollable state due to an erroneous detection by the information processing device 1 , winding in that state is not performed. Thereby, it is possible to prevent a large time loss due to failure to grasp garbage and the like.

The crane control part 301 will progress to the process of S27, when it determines with carrying out winding up in S25 (YES in S25), and will make the grab lifting control part 304 carry out the winding up of the grab bucket 934.

On the other hand, if the crane control unit 301 has not notified from the information processing device 1 that the grab bucket 934 has become a rollable state before the end time of the garbage picking operation started in S24, it is determined not to execute the operation based on the notification. Roll up (NO in S25). In this case, the crane control unit 301 waits until the garbage picking operation started in S24 is completed (S26), and then causes the grab lifting control unit 304 to roll up the grab bucket 934 (S27).

In this way, when the crane control unit 301 does not detect that the state of being able to be rolled up is not detected within a predetermined period of time after the start of the operation of grabbing garbage, the grab 934 is rolled up by the lapse of the predetermined period as a trigger. Accordingly, even when the information processing device 1 cannot detect that the roll-up state is enabled, the roll-up operation can be performed after a predetermined period of time has elapsed, thereby preventing time loss from occurring.

The above-mentioned predetermined period may be set as the period from the start of the predetermined garbage picking operation by the crane 93 to the end. This is because the garbage pick-up operation is a series of operations of the crane 93 that are prescribed to be able to roll up the grab bucket 934 , and it is highly likely that the grab bucket 934 will be in a rollable state at the end of the operation. For example, when the control of the garbage grabbing action is to execute the control of the specified number of times for the opening instruction and the closing instruction of the grab bucket 934 according to the specified time, the moment when the grab bucket 934 completes the closing action based on the specified number of closing instructions, It is judged that the above-mentioned prescribed period has passed.

In addition, the said predetermined period may be set based on the past control history of the crane 93, etc. FIG. In this case, it is possible to determine the time at which the grab bucket 934 is likely to be in a rollable state after the action of grabbing garbage is started based on the past control history, etc., and set the above-mentioned specified period. For example, it is possible to extract control histories in which appropriate rolling was performed without errors such as falling garbage during rolling, and determine the possibility of being in a rollable state based on the start time of rolling in these control histories higher moment. In addition, the predetermined time may be set based on a determined time. For example, the value obtained by adding a predetermined margin to the average time from the start of the operation of picking up the garbage to the start of rolling in the control history in which appropriate rolling is performed may be used as the predetermined time.

In S28, the crane control part 301 determines whether the closing degree of the grab bucket 934 is insufficient. Here, when the information processing device 1 notifies that the grab bucket 934 is in a half-open state, the crane control unit 301 determines that the closing degree of the grab bucket 934 is insufficient (YES in S28 ), and proceeds to the process of S29 . On the other hand, if the grab bucket 934 is not notified from the information processing device 1 that it is in the half-open state, the crane control unit 301 makes a negative determination in S28, and proceeds to the process of S30.

In S29 , the crane control unit 301 causes the grab bucket opening and closing control unit 303 to perform control to increase the closing degree of the grab bucket 934 . Thereby, since the grab 934 will be in the state which hold|maintains garbage more tightly, it can reduce the possibility that the held garbage will fall to an undesired position.

In S30, the crane control part 301 specifies the drop-off position of garbage. Then, the grab movement control unit 302 is instructed to move the grab bucket 934 to the above-mentioned drop position, and then the grab bucket opening and closing control unit 303 is instructed to bring the grab bucket 934 into an open state. Thereby, the garbage picked up by the garbage picking position is dropped to the dropping position, and the process of FIG. 11 is complete|finished. In addition, the dropping position may be a position in a garbage pit or a position in a hopper.

(Experiment for effect verification)

An experiment was conducted to verify that by rolling up the grab bucket 934 at the timing when the information processing device 1 detects the off-the-ground state, it is possible to reduce the time loss from when the roll-up becomes possible to when the roll-up is performed. In this experiment, a comparison was made between the time when the information processing device 1 detected the off-the-ground state and the time when a predetermined garbage picking operation was completed.

The results of the above experiments are shown in FIG. 12 . FIG. 12 is a diagram showing the transition of the control history for the crane 93 and the results of status determination by the information processing device 1 . More specifically, the control histories of the wind-up command and the open-close command of the grab bucket 934 are shown in the upper graph of FIG. 12 . In addition, the transition of the result of the state judgment of the grab bucket 934 and the transition of the measurement result of the weight of the garbage picked up by the grab bucket 934 are shown in the lower graph of FIG. In addition, in determining the state of the grab 934 , it is determined which of the three states, the airborne state, the grounded state, and the off-the-ground state. In addition, in the graph, the airborne state is represented by a numerical value 3, the grounded state is represented by a numerical value 1, and the grounded state is represented by a numerical value 2. If the state cannot be determined (all determination results are less than the threshold value), 0 is set.

At 21:53:45 immediately after the start of the experiment, the grab bucket 934 was in the air in a state of not grabbing garbage. After that, the grab bucket 934 is temporarily closed and then opened again, and at around 21:54:02, a negative roll-up command, that is, a control to lower the grab bucket 934 is executed. The grab weight shown in the lower graph of FIG. 12 reflects this control result, and is 0 while the grab 934 is in the air, and is a negative value when the grab 934 is grounded.

Next, the rubbish picking operation was performed for about 20 seconds from 21:54:15 to 21:54:35 after the grab bucket 934 was lowered. Specifically, at 21:54:15, the negative opening and closing command, that is, the control of closing the grab bucket 934 was executed, and then the closing degree of the grab bucket 934 was returned to the neutral position (0) and repeated twice at a low speed for a short time. The rolled up control. This roll-up makes the grab 934 slightly suspended from the surface of the garbage. As shown in the lower graph of FIG. In addition, these controls are controls for bringing the grab bucket 934 into a ground-off state.

And, at 21:54:35 when the garbage picking operation was performed, the rolling up of the grab bucket 934 is started. In addition, the rolling up at this time is performed until the grab bucket 934 reaches the height which can move in the horizontal direction, and the rolling speed is faster than the rolling up at the time of garbage picking operation, and the rolling distance is also long.

As shown in the lower graph of FIG. 12 , the result of the state judgment of the grab bucket 934 during the above-mentioned control period, there is an erroneous judgment and a state where judgment cannot be made around 21:53:55 to 21:54:00. . However, the result of this state determination can accurately determine the state of the grab bucket 934 as a whole. In particular, it is possible to accurately determine the timing of the transition from the airborne state to the grounded state around 21:54:12, and the transition from the grounded state to the liftoff state around 21:54:30.

Here, the time of 21:54:30 at which the transition from the grounded state to the grounded state is detected is about 5 seconds earlier than the time of 21:54:35 at which the garbage picking operation ends and the rolling up starts. In other words, in the automatic control shown in FIG. 12 , although the roll-up is possible at 21:54:30, it takes about 5 seconds until the garbage pick-up operation is continued afterwards. loss.

The results of this experiment show that the information processing device 1 can reduce the time loss from when the roll-up becomes possible to when the roll-up is performed by starting the roll-up when the transition to the off-the-ground state is detected.

(Modification)

The subject of execution of each process described in the above embodiment is an example and can be changed as appropriate. For example, it is also possible to make a plurality of computers execute the learning method shown in FIG. 9 , the information processing method shown in FIG. 10 , and the control method shown in FIG. 11 .

For example, the processes of S1 to S6 in FIG. 9 can also be realized by a learning system including a computer that executes the processes of S1 to S4 in FIG. 9 and a computer that executes the processes of S5 and S6. In addition, for example, each processing of S12 to S14 in FIG. 10 may be executed by a separate computer, or these processings may be performed by the control device 3 .

In addition, the components of the control system 100 are not limited to the example of the above-mentioned embodiment, either. For example, as described above, when the control device 3 performs each process of S12 to S14 in FIG. 10 , the information processing device 1 is omitted from the components of the control system 100 . In addition, the construction of a model such as a state determination model may be performed outside the control system 100 , and in this case, the learning device 5 is omitted from the components of the control system 100 . In addition, the imaging device 7 can be excluded from the components of the control system 100 because it provides images for control and has nothing to do with the control itself. In this case, the image captured by the imaging device 7 outside the control system 100 is acquired and controlled. In addition, the crane 93 whose operation is controlled by the control system 100, and the waste incineration facility equipped with the crane 93 are also included in the scope of the present invention.

In addition, as described above, the results of determination and detection by the information processing device 1 can be used for automatic control, and can also be used as reference information for manual control. In the latter case, for example, an output device such as a display device may output the result of determination and detection by the information processing device 1 and present it to the operator of the crane 93 . Accordingly, the operator of the crane 93 can manually control the crane 93 as a reference.

(Software-based implementation example)

The control blocks of the information processing device 1, the control device 3, and the learning device 5 (each part included in the control parts 10, 30, 50) can be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or It can be realized by software.

In the latter case, the information processing device 1 , the control device 3 , and the learning device 5 include a computer that executes commands of a program that is software that realizes each function. This computer includes, for example, one or more processors, and includes a computer-readable storage medium storing the above-mentioned program. In addition, in the computer described above, the processor reads and executes the program from the storage medium, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit: Central Processing Unit) can be used. As the above-mentioned storage medium, "non-transitory tangible medium" can be used. For example, in addition to ROM (Read Only Memory: read-only memory), magnetic tape, optical disk, card, semiconductor memory, programmable logic circuit, etc. can also be used. Wait. In addition, a RAM (Random Access Memory: Random Access Memory) for expanding the above program may be provided. In addition, the above-mentioned program may be provided to the above-mentioned computer via any transmission medium (communication network, radio wave, etc.) capable of transmitting the program. In addition, one aspect of the present invention can also be implemented in the form of a data signal embodied by electronic transmission of the above-mentioned program and embedded in a carrier wave.

The present invention is not limited to the above-described embodiments, and various changes can be made within the scope of the claims. Embodiments obtained by appropriately combining technical solutions disclosed in different embodiments are also included in the technology of the present invention. within range.

Explanation of reference signs

1-information processing device; 101-image acquisition unit; 105-ground lift detection unit (detection unit); 3-control device; 7-photography device; 100-control system; 93-crane; 934-grab bucket.

Claims (7)

1. A control system comprising: A photographic device that photographs a series of states of a crane carrying garbage after grabbing garbage with a grab bucket to rolling up the grab bucket in the state of grabbing garbage; An information processing device that detects that the grab bucket is in a state where it is possible to roll up the grab bucket by inputting an output value obtained by the photographed image to a state judgment model, wherein the state judgment model An image of the grab bucket in the state of being littered and able to roll up was constructed as machine learning from the teacher data; and A control device that causes the crane to perform a wind-up operation of the grab bucket when the information processing device detects that the grab bucket can be rolled up. 2. The control system of claim 1, wherein: When the control device does not detect that the grab is in a rollable state within a predetermined period of time after the start of the operation of grabbing the garbage with the grab, the grab is performed when the predetermined period has elapsed. The rolling action of the bucket. 3. The control system according to claim 1 or 2, characterized in that, The control device performs the rolling-up operation of the grab bucket on the condition that a predetermined lower limit period has elapsed since the grabbing operation of the grab bucket starts and the grab bucket is in a rollable state. . 4. A control system according to any one of claims 1 to 3, characterized in that, the information processing device detects, based on the image, that the grab of the crane on the surface of the refuse is tilted, When the information processing device detects that the grab bucket is tilted, the control device performs control to change the grabbing position of the garbage by the crane. 5. A control system according to any one of claims 1 to 4, characterized in that, The information processing device detects, based on the image, that the grab is not closed enough to hold the garbage, The control device performs control to close the grab bucket when the information processing device detects that the grab bucket is insufficiently closed to hold the garbage. 6. An information processing device comprising: an image acquiring unit that acquires images of a series of states of a crane that transports garbage, after grabbing garbage with a grab bucket, to rolling up the grab bucket in a state of grabbing garbage; and A detecting unit that detects that the grab bucket can be rolled up based on an output value obtained by inputting the image to a state judgment model, wherein the state judgment model is based on the fact that the grab bucket has been picked up and can be rolled up. The image of the rolled-up grab bucket was constructed as machine learning from the teacher data. 7. An information processing method using an information processing device, comprising: An image acquisition step of acquiring images of a series of states of a crane carrying garbage after grabbing the garbage with the grab until the grab is rolled up in a state of grabbing the garbage; and A detecting step of detecting that the grab bucket can be rolled up based on an output value obtained by inputting the image to a state judgment model, wherein the state judgment model detects that the grab bucket has been picked up and can be rolled up. The image of the rolled-up grab bucket was constructed as machine learning from the teacher data.

Images