JP6600726B1 - Determination apparatus and determination method - Google Patents

Abstract

It is determined whether or not cargo collapse occurs. A determination device (1) is generated by machine learning using an image obtained by photographing a load immediately before the load collapse when the load to be transported by a forklift (3) occurs. A load collapse determination unit (103) that determines whether or not a load collapse occurs in the target load from an image obtained by photographing the target load to be determined using the learned model (102). [Selection] Figure 1

Description

  The present invention relates to a determination device that determines the possibility of collapse of a load to be transported by a work vehicle.

  In a cargo handling operation using a work vehicle such as a forklift, a load placed on a pallet is transported or unloaded together with the pallet. Since the load is placed on the pallet, the collapse of the load may occur due to an external force applied during the work. Further, even when the work vehicle is not working on the load, the collapse of the load may occur due to vibration of the floor or shelf on which the load is placed.

  As a technique for preventing the occurrence of load collapse, for example, the following Patent Document 1 can be cited. This document describes a safety device that determines whether or not the height of a package to be transported is a height that can be transported without collapsing the cargo before it is transported.

JP 2008-156093 A

  In Patent Document 1 described above, the height of a load specified using an ultrasonic sensor is used as a criterion for determining whether or not a load collapse occurs. However, cargo collapse does not necessarily occur if the packages are stacked high. In addition, it is a matter that the driver can easily recognize visually that the luggage is piled up high, and there is little need to judge using a safety device.

  An object of one embodiment of the present invention is to realize a determination device and the like that can determine whether or not collapse of a target load occurs based on an image obtained by capturing the target load.

  In order to solve the above-described problem, the determination apparatus according to one aspect of the present invention captures an image of a load immediately before the load collapse when the load to be transported by the work vehicle is collapsed. A load collapse determination unit that determines whether or not a load collapse occurs in the target load from an image obtained by photographing the target load to be determined using a learned model generated by machine learning using ing.

  Moreover, in order to solve said subject, the determination method which concerns on 1 aspect of this invention is the determination method by a determination apparatus, Comprising: When load collapse generate | occur | produces in the load used as the object conveyed by the working vehicle Whether the target load collapses from the image of the target load to be judged using the learned model generated by machine learning using the image of the load immediately before the load collapse A load collapse determination step for determining whether or not is included.

  According to one aspect of the present invention, it is possible to determine whether or not collapse of a target load occurs based on an image obtained by capturing the target load. Accordingly, it is possible to prevent the occurrence of the collapse of the cargo by warning the operator of the work vehicle when the possibility of the collapse of the cargo is high.

It is a block diagram which shows an example of the principal part structure of the determination apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline | summary of the determination system containing the said determination apparatus. It is a flowchart which shows an example of the process which the said determination apparatus performs. It is a figure explaining the example of the learned model which the determination apparatus concerning other embodiment of this invention uses. It is a block diagram which shows an example of the principal part structure of the determination apparatus which concerns on further another embodiment of this invention.

Embodiment 1
(System overview)
An outline of the determination system of this embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an outline of the determination system 100. The determination system 100 is a system that determines whether or not load collapse occurs in the load conveyed by the work vehicle. In the present embodiment, an assembly of a plurality of loads stacked on one pallet is referred to as “load”.

  A determination system 100 illustrated in FIG. 2 includes a determination device 1 that performs the above determination, a photographing device 2 that captures the state of the load, and a forklift 3 that is an example of a work vehicle. In the example of FIG. 2, the imaging device 2 captures an image of a worker operating the forklift 3 to convey a load. In addition, the imaging device 2 should just be arrange | positioned in the position which can image | photograph a load. For example, the imaging device 2 may be mounted on the forklift 3. Moreover, you may divert the imaging device currently used for the other use as the imaging device 2. FIG. For example, when a surveillance camera is installed in a facility where a load is placed, it may be used as the imaging device 2. When a drive recorder is mounted on the forklift 3, the camera is used as the imaging device 2. It may be used as

  The image captured by the imaging device 2 is transmitted to the determination device 1, and the determination device 1 determines whether or not cargo collapse occurs from the state of cargo in this image. And when the determination apparatus 1 determines that cargo collapse occurs, it notifies the operator who drives the forklift 3 to that effect. It should be noted that the image captured by the image capturing device 2 may be any image that has an appearance characteristic peculiar to a load that is highly likely to collapse, and may be a still image or a moving image. There may be.

  Thus, according to the determination system 100, the determination apparatus 1 can determine whether or not cargo collapse occurs. And the determination apparatus 1 can alert | report to a worker that possibility that a load collapse will occur before a load collapse occurs. Accordingly, it is possible to prevent the worker from collapsing due to the cargo handling operation performed on the cargo that is highly likely to collapse. It is also possible to prevent the occurrence of load collapse by detecting a load with high possibility of load collapse and taking measures such as reloading so as not to cause the load collapse.

  In addition, although this embodiment demonstrates the example which mounted the determination apparatus 1 in the forklift 3, the determination apparatus 1 may be installed in the position away from the forklift 3. FIG. In this case, the determination device 1 notifies that there is a load that is likely to cause a load collapse by communicating with the forklift 3 or communicating with a terminal device possessed by the driver of the forklift 3 or the like. In this case, it is also possible to monitor a load within a range photographed by one or a plurality of photographing devices 2 by one determination device 1. As a result, when the determination device 1 determines that there is a load with a high possibility of collapse of cargo, the determination device 1 alerts the forklift 3 that intends to perform a cargo handling operation on the load, or a person is involved in the load. You can also warn you not to get close.

(Main components of the judgment device)
A main configuration of the determination apparatus 1 will be described with reference to FIG. FIG. 1 is a block diagram illustrating an example of a main configuration of the determination apparatus 1. As illustrated, the determination device 1 includes a control unit 10 that controls each unit of the determination device 1 and a storage unit 20 that stores various data used by the determination device 1. Further, the determination device 1 includes a communication unit 30 for the determination device 1 to communicate with an external device (for example, the imaging device 2), and an output unit 40 for the determination device 1 to output information. The control unit 10 includes an input data generation unit 101, a learned model 102, a load collapse determination unit 103, and a notification unit 104.

  The input data generation unit 101 acquires an image of a target load that is a target for determining whether or not a load collapse has occurred, and generates input data for the learned model 102. More specifically, the input data generation unit 101 acquires an image of the target cargo captured by the imaging device 2 from the imaging device 2, and generates input data for the learned model 102 from the acquired image. Details of the input data will be described later.

  The learned model 102 is a learned model generated by machine learning using, as teacher data, an image obtained by photographing a load immediately before the load collapses when the load collapses. When the input data generated by the input data generation unit 101 is input to the learned model 102, the learned model 102 outputs information (for example, a probability value) indicating the possibility of cargo collapse.

  The cargo collapse determination unit 103 determines whether or not cargo collapse occurs based on the output data from the learned model 102. For example, the load collapse determination unit 103 determines that the load collapse occurs if the probability value indicated by the output data of the learned model 102 is equal to or greater than a predetermined threshold. The probability value is a value indicating the probability that the cargo collapse will occur (more precisely, a value indicating the probability that the photographed load state corresponds to the state where the cargo collapse occurs). Note that the load collapse determination unit 103 may use the probability value as a determination result as to whether or not load collapse occurs, instead of performing the determination based on the threshold value.

  The notification unit 104 notifies that there is a possibility that the load may collapse. Specifically, the notification unit 104 controls the output unit 40 to output information indicating that there is a high possibility that an operation error will occur. In the present embodiment, since it is assumed that the determination device 1 is mounted on the forklift 3, by notifying the operator who operates the forklift 3 by causing the output unit 40 to output the above information. Can do. The output mode of the information is not particularly limited. For example, if the output unit 40 is a display unit, the information may be displayed and output if the output unit 40 is an audio output unit. Further, when the determination device 1 is located away from the forklift 3, the notification unit 104 may cause the output unit included in the forklift 3 to output the information by communicating with the forklift 3. In addition, for example, the notification unit 104 outputs the above information to another device such as a terminal device possessed by the worker or a terminal device of an administrator who manages the work or cargo of the worker. You may go.

(About learned models)
The learned model 102 of the present embodiment is a learned model generated by supervised machine learning so that it can be determined whether or not cargo collapse occurs. As the teacher data for generating the learned model 102, for example, an image obtained by photographing the cargo immediately before the collapse of the load when the collapse of the load has occurred in the past can be used. Accordingly, it is possible to generate a learned model 102 that outputs a probability value that a load collapse occurs when input data generated from an image obtained by photographing a load is input.

  In addition, the trigger of the load collapse is not particularly limited. For example, the load collapse may occur naturally when the cargo handling operation is not performed, or the cargo collapse may be triggered by the cargo handling operation by the work vehicle. Also good. However, it is preferable not to use an image taken in a case where the cause of the collapse of the load is not in the way of loading the load but is in the worker (for example, a case where the load collapses due to an operation error) as the teacher data. .

  Further, an image obtained by photographing a load in which no cargo collapse has occurred can also be used as teacher data. In this case, it is possible to generate the learned model 102 that outputs the probability that the load collapse occurs and the probability that the load collapse does not occur when input data generated from an image obtained by photographing the load is input. The probability value of the former increases as the appearance of the load has the same characteristics as the appearance of the load that has collapsed.The probability value of the latter indicates that the appearance of the load has collapsed. The higher the value, the more the same characteristics as the appearance of the unloaded cargo.

  When using an image as the teacher data, it is preferable to use, as the learned model 102, for example, a CNN (Convolutional Neural Network) suitable for computation using the image as input data. When the image used as the teacher data is a moving image, the learned model 102 is, for example, a learned model that combines CNN and RNN (Recurrent Neural Network) or CNN and LSTM (Long Short-Term Memory). It is preferable to use it. Learned to output highly reliable output data for input data that is moving images by combining a model suitable for handling time-series data such as RNN and LSTM and CNN with high image recognition performance The model 102 can be generated.

(About input data)
The input data generation unit 101 generates input data to be input to the learned model 102 generated by performing machine learning using a large number of teacher data as described above. Specifically, the input data generation unit 101 generates the input data from an image obtained by photographing a target cargo that is a target for which the forklift 3 performs a cargo handling operation.

  The image used for the input data is preferably an image of the target load under the same shooting conditions as the image used for the machine learning teacher data of the learned model 102. The photographing conditions include, for example, the positional relationship between the photographing device 2 and the load, the ambient brightness, and the like. In order to align the shooting conditions, for example, teacher data is generated from an image captured by the imaging device 2 fixed at a position where the load enters the angle of view, and an image obtained by capturing the target cargo by the same imaging device 2 is used as input data. preferable.

(Process flow)
A flow of processing executed by the determination apparatus 1 will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of a process (determination method) executed by the determination apparatus 1. Note that, as in the example of FIG. 2, an image obtained by photographing the target load by the photographing device 2 is transmitted to the determination device 1.

  In S <b> 1, the input data generation unit 101 acquires an image of the target cargo captured by the imaging device 2. The opportunity for image acquisition is not particularly limited. For example, as shown in FIG. 2, when the target load is photographed by the photographing device 2 whose position is fixed, the input data generation unit 101 includes the forklift 3 in the photographing range. This may be detected by a sensor or the like, and an image may be acquired in response to the detection. In addition, the input data generation unit 101 may acquire an image, for example, when the forklift 3 starts a cargo handling operation for the target load. The fact that the cargo handling operation has started includes, for example, that the forklift 3 has directly faced the target load, that the fork has moved up and down, or that the operation for causing the forklift 3 to face the load or the operation for raising and lowering the fork has been performed. It can be identified by detection. Further, the input data generation unit 101 may cause the imaging device 2 to start imaging at these triggers, and may control the imaging device 2 to transmit the captured image to the input data generation unit 101.

  In S2, the input data generation unit 101 generates input data of the learned model 102 from the image acquired in S1. In S3, the input data generation unit 101 inputs the input data generated in S2 to the learned model 102, and the learned model 102 has a probability that a load collapse will occur in the target cargo shown in the image acquired in S1. Output.

  In S4 (load collapse determination step), the load collapse determination unit 103 determines whether load collapse occurs from the output data of the learned model 102. If the cargo collapse determination unit 103 determines that cargo collapse occurs (YES in S4), the process proceeds to S5. On the other hand, when the load collapse determination unit 103 determines that no load collapse occurs (NO in S4), the process of FIG. 3 ends.

  In S <b> 5, the notification unit 104 notifies the operator who operates the forklift 3 that there is a high possibility that the cargo collapse will occur, and thus the processing of FIG. 3 ends. Since the notification mode is as described above, the description will not be repeated here. In addition, the alerting | reporting part 104 may alert | report in a different aspect according to a probability so that an operator's attention may be called more strongly, so that the probability which the output data of the learned model 102 shows is high. For example, when the notification unit 104 performs notification by voice, the volume may be increased as the probability increases.

(Summary of Embodiment 1)
As described above, the load collapse determination unit 103 of the determination apparatus 1 according to the present embodiment has photographed the load immediately before the load collapse when the load to be transported by the work vehicle is collapsed. Using a learned model 102 generated by machine learning using an image, it is determined whether or not a load collapse occurs in the target load from an image obtained by capturing the target load as a determination target.

  Thereby, based on the image which image | photographed the target load, it can be determined whether the collapse of the target load occurs. In addition, when it is determined that cargo collapse occurs, the notification unit 104 notifies the worker, so that it is possible to prevent the cargo collapse from occurring.

[Embodiment 2]
Another embodiment of the present invention will be described below. For convenience of explanation, members having the same functions as those described in the first embodiment are given the same reference numerals, and the description thereof will not be repeated. The same applies to the third and subsequent embodiments.

  When a work is performed on a load using a work vehicle, an external force specific to the work is applied to the load. For example, in the case of an operation for lifting a load, a vertical upward external force is applied to the lifted load. Further, for example, in the case of an operation of stacking another load or load on a certain load, an external force vertically downward is applied to the certain load. For this reason, it may happen that a load that has not collapsed in one operation will collapse in another operation.

  Therefore, the determination device 1 according to the present embodiment provides a load immediately before a predetermined operation when load collapse occurs due to an external force applied to the load due to the execution of the predetermined operation using the work vehicle. A learned model 102 generated by machine learning using an image obtained by shooting is used. And the load collapse determination part 103 of this embodiment determines whether load collapse will generate | occur | produce at the time of the said predetermined | prescribed operation | work from the image which image | photographed the object load. As a result, it is possible to determine with high accuracy whether or not a load collapse has occurred in consideration of how to apply an external force peculiar to a predetermined work. In addition, since it is possible to grasp that there is a risk of cargo collapse before a predetermined operation, it is possible to prevent the occurrence of cargo collapse.

  The learned model 102 used by the determination apparatus 1 of this embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of the learned model 102 that is used by the determination apparatus 1 of the present embodiment.

  FIG. 4A shows a state in which the work of stacking the load B on the load A is performed using a forklift. In such an operation, when the loading method of the load A is a weak loading method against the vertically downward external force, the vertically downward external force applied by the load B may cause the load A to collapse.

  For this reason, the determination apparatus 1 of the present embodiment may use the learned model 102 for the stacking operation when determining whether or not the load collapse has occurred in the stacking operation. Such a learned model 102 can be generated by machine learning using, as teacher data, an image obtained by photographing a load immediately before the load collapse when the load collapse occurs in the stacking operation.

  On the other hand, FIG. 4B shows a state where the work of lifting the load C is performed using a forklift. In such an operation, when the loading method of the load C is a weak loading method with respect to the vertically upward external force, the load C may collapse due to the vertically upward external force applied by the forklift.

  For this reason, the determination apparatus 1 of the present embodiment may use the learned model 102 for lifting work when determining whether or not load collapse has occurred in the lifting work of the load. Such a learned model 102 can be generated by machine learning using, as teacher data, an image obtained by photographing a load immediately before the load collapse when the load collapse occurs in the lifting operation.

  Further, a learned model 102 for each work as described above may be prepared in advance. Then, the load collapse determination unit 103 may select the learned model 102 corresponding to the work to be performed by the forklift 3, and determine whether or not the load collapse has occurred based on the output data of the learned model 102. Thereby, also when performing the some operation | work from which how external force is applied with respect to object load is performed, the presence or absence of occurrence of load collapse can be determined accurately.

  In addition, the method for specifying the work to be performed by the forklift 3 is not particularly limited. For example, a configuration in which an operator inputs a work to be performed to the determination device 1 or a configuration in which the load collapse determination unit 103 identifies by analyzing an image obtained by photographing at least one of the forklift 3 and the worker is possible. Also good.

[Embodiment 3]
Still another embodiment of the present invention will be described below. The determination apparatus 1 according to the present embodiment is different from the determination apparatuses 1 according to the above-described embodiments in that the presence / absence of load collapse is determined in consideration of the weight of the load.

  Even if the appearance of the load is the same, the possibility of collapse of the load may be high or low depending on the weight of the load. For example, if the weight of the load is heavy, there is a high possibility that the load will not collapse even if the stacked loads are slightly deviated, but if the load is light, there is a possibility that the load collapses with a slight deviation.

  Therefore, in this embodiment, a plurality of learned models 102 corresponding to the weight of the load are prepared in advance. And the load collapse determination part 103 of this embodiment determines whether load collapse occurs using the learned model 102 which changes according to the weight of object load. This makes it possible to perform more accurate determination in consideration of the weight of the target load.

  For example, by classifying a load by its weight and using the teacher data as an image of the load immediately before the load collapse when the load collapses in the load of each classification, a learned model for each classification 102 may be generated. When these learned models 102 are used, the load collapse determination unit 103 identifies the classification of the target load and determines whether or not load collapse has occurred based on the output data of the learned model 102 corresponding to the classification.

  In addition, the method of specifying the weight (or classification based on the weight) of the target load is not particularly limited. For example, the operator may input the weight or classification into the determination device 1 or the load collapse determination unit 103 may specify the information by reading the information from a recording medium on which information such as the weight of the target load is recorded. It is good also as a structure. For example, the load collapse determination unit 103 may read information such as the weight from an RF tag attached to the target load by RFID (Radio Frequency Identification).

[Embodiment 4]
Still another embodiment of the present invention will be described below. In addition to the processing in the above-described embodiment, the determination device 1 according to the present embodiment performs processing for generating teacher data from an image acquired from the imaging device 2 and updating the learned model.

  Teacher data generation and learning model update will be described with reference to FIG. FIG. 5 is a diagram illustrating a main configuration of the determination apparatus 1 according to the fourth embodiment of the present invention. As shown in FIG. 5, the control unit 10 of the determination apparatus 1 of the present embodiment includes a load collapse presence / absence determination unit 107, a teacher data generation unit 105, and a learning unit 106.

  The load collapse presence / absence determination unit 107 determines whether or not an actual load collapse has occurred in the target load after the load collapse determination unit 103 determines whether or not the load collapse has occurred. Then, the teacher data generation unit 105 associates the input data input to the learned model 102 for determination by the load collapse determination unit 103 with the determination result of the load collapse presence / absence determination unit 107 to associate the learned data with the learned model 102. Generate teacher data for machine learning. The input data is generated by the input data generation unit 101.

  According to the above configuration, since teacher data is generated by automatically determining whether or not cargo collapse has actually occurred, teacher data can be generated without bothering the user of the determination device 1. . Note that the method for determining whether or not the cargo collapse has actually occurred is not particularly limited. For example, when a load collapse occurs, the appearance of the target load changes greatly from that before the load collapse occurs. Therefore, the load collapse presence / absence determination unit 107 analyzes the image obtained by photographing the target load to actually cause the load collapse. It can also be determined whether or not. The teacher data generation unit 105 may generate the teacher data only when the determination result of the load collapse presence / absence determination unit 107 is a determination result that the load collapse has occurred.

  In addition, the learning unit 106 updates the learned model 102 using the teacher data generated by the teacher data generation unit 105. As described above, in the determination apparatus 1 of the present embodiment, teacher data is generated every time the load collapse determination unit 103 performs determination, and the learned model 102 is updated using the teacher data. Thereby, the determination accuracy of the determination apparatus 1 can be maintained or improved while using the determination apparatus 1.

[About distributed processing]
A part of the processing executed by the determination apparatus 1 described in each of the above embodiments may be executed by one or a plurality of apparatuses connected to the determination apparatus 1 in communication. For example, the process executed by the learned model 102 may be executed by an AI server that is connected to the determination apparatus 1 in communication. In this case, the determination apparatus 1 generates input data from the image captured by the imaging apparatus 2 and transmits the input data to the AI server, and receives the output data from the AI server to determine whether the load may collapse Judgment is made.

[About input data]
As input data for the learned model 102, an image acquired from the photographing apparatus 2 may be used as it is, or an image obtained by performing a predetermined process on the image may be used. The predetermined process may be any process that can reduce information unrelated to the feature points without losing the feature points in the image. For example, if the image is a color image, the input data may be converted to gray scale. In addition, it is possible to extract only a moving region from the image as input data.

  Further, for example, a load may be detected from an image, and only an area where the detected load is reflected may be input data. Thereby, since the influence of the background of a load can be excluded, determination accuracy can be improved. Note that, for example, a learned model such as CNN can be used to detect the load.

[Example of software implementation]
The control block (particularly, each unit included in the control unit 10) of the determination apparatus 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software. .

  In the latter case, the determination apparatus 1 includes a computer that executes instructions of a program that is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. For example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) can be used as the processor. As the recording medium, a “non-temporary tangible medium” such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the program may be further provided. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Determination apparatus 102 Trained model 103 Unloading determination part 105 Teacher data generation part 107 Unloading presence determination part

Claims (4)

When a load collapse occurs in the load to be transported by the work vehicle, it is determined using a learned model generated by machine learning using an image of the load immediately before the load collapse. A load collapse determination unit that determines whether or not a load collapse occurs in the target load from an image obtained by capturing the target load ,
The learned model captures a load immediately before the predetermined operation when a load collapse occurs due to an external force applied to the load due to the predetermined operation performed using the work vehicle. Generated by machine learning using the image,
The load collapse determination unit determines whether or not load collapse occurs during the predetermined work from an image obtained by photographing the target load . The determination apparatus according to claim 1 , wherein the load collapse determination unit determines whether load collapse occurs using a learned model that differs depending on a weight of the target load. After the determination by the load collapse determination unit, a load collapse presence / absence determination unit that determines whether or not the actual load collapse has occurred in the target load,
Generates teacher data for machine learning of the learned model by associating the input data input to the learned model for determination by the load collapse determination unit with the determination result of the load collapse presence determination unit determination apparatus according to claim 1 or 2, characterized in that it and a supervisor data generator for. A determination method by a determination device,
When a load collapse occurs in the load to be transported by the work vehicle, it is determined using a learned model generated by machine learning using an image of the load immediately before the load collapse. the target load from the captured image composed, viewed including the collapsing determination step of determining whether collapsing occurs to the subject load,
The learned model captures a load immediately before the predetermined work when a load collapse occurs due to an external force applied to the load due to the predetermined work being performed using the work vehicle. Generated by machine learning using the image,
In the load collapse determination step, it is determined whether or not load collapse occurs during the predetermined work from an image obtained by photographing the target load .

Images