JP7462195B2 - ROBOT INTERFERENCE DETECTION DEVICE, ROBOT INTERFERENCE DETECTION METHOD, ROBOT CONTROL DEVICE, ROBOT CONTROL SYSTEM, HUMAN OPERATION PREDICTION DEVICE, AND HUMAN OPERATION PREDICTION METHOD - Google Patents

Description

The present invention relates to a robot interference detection device that detects the possibility of contact between a robot and a detection target such as a worker in an environment where the two coexist.

Conventionally, various attempts have been known to prevent accidental contact between robots and workers at production sites such as factories. For example, a method is known in which a safety fence or the like is used to separate the human work area from the robot work area, but this method does not allow for collaborative work between humans and robots, resulting in low production efficiency and space efficiency. For example, Patent Documents 1 and 2 listed below disclose attempts to monitor for the intrusion of humans into a predefined area around the robot and control the robot's operation or limit the robot's torque.

JP 2010-208002 A German Patent Publication No. DE 102016222245 A1

In the first place, in learning algorithms that use neural networks, etc. (e.g., deep learning, etc.), in order to update the trained model through additional learning, it is necessary to manually adjust the learning parameters. Even when a worker (person) repeatedly performs the same task, the worker's movement pattern may change depending on the worker's fatigue, etc. Therefore, in order to make highly accurate predictions, it is necessary to update the trained model. However, when using a learning algorithm that requires manual parameter adjustment, it is difficult to properly update the trained model in real time.

One aspect of the present invention was made in consideration of the above problems, and aims to automatically update a trained model in real time and predict human behavior.

In order to solve the above problems, a robot interference determination device according to one embodiment of the present invention includes an object state acquisition unit that acquires information indicating the change in the position of an object present within a predetermined area up to the present time, a robot state acquisition unit that acquires a predicted position and predicted velocity vector of a predetermined moving part of the robot after a predetermined time, a prediction unit that calculates a predicted position and predicted velocity vector of the object after the predetermined time based on the information indicating the change in the position of the object up to the present time, and an interference determination unit that determines in real time whether or not there is a possibility of contact between the object and the moving part based on the predicted position and predicted velocity vector of the object and the predicted position and predicted velocity vector of the moving part.

According to the above configuration, the robot interference determination device acquires information indicating the change in the position of the object up to the present time, and calculates the predicted position and predicted velocity vector of the object using the information indicating the change in the position up to the present time. The information indicating the change in the position of the object up to the present time is, for example, the position of the object at the present time (current position) and the velocity vector of the object at the present time (current velocity vector). In addition, the various pieces of information shown as input in the table of FIG. 15 described later are each examples of information indicating the change in the position of the object up to the present time.

The robot interference determination device then determines in real time whether or not there is a possibility of contact between the object and the movable part based on the calculated predicted position and predicted velocity vector of the object and the predicted position and predicted velocity vector of the movable part.

The robot interference determination device can avoid interference between the robot and the object without using a safety fence, thereby realizing space savings in production sites, etc.

In addition, the robot interference determination device determines the possibility of interference between the robot and the object at the specified time (i.e., in the future) at the present time, so even if it determines that there is a possibility of interference, it can currently execute processing to avoid the interference with ample time.

Here, if the device that currently determines the possibility of interference determines that there is a possibility of interference at the present time, then in order to avoid the current interference, the robot will need to suddenly decelerate and stop, among other actions, which will inevitably result in a decrease in production efficiency.

In contrast, the robot interference determination device determines the possibility of future interference at the present time, and therefore the robot's movements required to avoid interference can be made smoother and to a minimum, compared to when the possibility of current interference is determined at the present time. In other words, the robot interference determination device can prevent interference between the robot and the object, while suppressing a decrease in production efficiency due to sudden deceleration and sudden stopping of the robot, etc.

The robot interference determination device therefore has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the robot and the object, such as a worker, at a production site such as a factory.

As mentioned above, the object may be, for example, a person such as a worker, and more specifically, may be an entire person or part of a person, or may be a non-human object (for example, a mobile robot). If the object is an entire person, the position of the object is, for example, the position of the center of gravity of the person. If the object is part of a person or a non-human object, the position of the object is the position of a part of the person, or the position of the non-human object.

The prediction unit may calculate the predicted velocity vector of the object using the results of learning the history of the object's position and velocity vector at each point in time, or it may use simple prediction, i.e., assume that the current velocity vector will continue.

In a robot interference detection device according to one aspect of the present invention, the robot may be equipped with a multi-joint manipulator in which multiple arms are connected by multiple joints, and the movable part may be one or more predetermined parts of the multi-joint manipulator.

According to the above configuration, the robot interference determination device currently determines the possibility of interference between the robot, which is equipped with a multi-joint manipulator in which multiple arms are connected at multiple joints, and the target object after the specified time (i.e., in the future).

The robot interference determination device therefore has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the target object, such as a worker, and the robot equipped with a multi-joint manipulator in a production site such as a factory.

In a robot interference determination device according to one aspect of the present invention, the interference determination unit may determine that there is a possibility of contact between the object and the moving part if the distance between the estimated position of the object and the estimated position of the moving part during the time from when the predetermined time has elapsed from the current time until the moving part stops moving after the movement stopping operation of the moving part starts is within a predetermined value.

According to the above configuration, the robot interference judgment device determines that there is a possibility of contact _between the object and the moving part when the distance between the estimated position of the object and the estimated position of the moving part is within a predetermined value from the point ( _t0 + Δt) when the predetermined time (Δt) has elapsed from the current time (t0) to the time when the movement of the moving part stops.

In other words, the robot interference determination device makes the above-mentioned determination using not only the distance between the estimated position of the object and the estimated position of the movable part at the time when the movement of the movable part stops, but also the distance between the estimated position of the object and the estimated position of the movable part at each point along the way.

Therefore, the robot interference judgment device has the effect of being able to make judgments that take into account the possibility of contact at each point from the point when the predetermined time (Δt) has elapsed from the current point ( _t0 ) (that is, the point _t0 + Δt) until the movement of the movable part stops.

Here, in 5.5.4.2.3 of ISO/TC15066, the safety distance Sp is defined as "Sp( _t0 ) = _Sh + _Sr + _Ss + c + _Zd + _Zr ". In the above standard, if the actual separation distance is equal to or less than the safety distance Sp, it is determined that there is a possibility of inadvertent interference. This can be interpreted as determining that there will be interference between a person and a robot when the distance between the estimated position of the person (the position where the person is estimated to be present) and the estimated position of the robot (the position where the robot is estimated to be present) at the time when the robot stops moving is equal to or less than a predetermined value (c + Zd + Zr).

In the above definition formula, "S _h " indicates "the distance a person moves from the current time to the time when the robot stops moving", and "S _r " indicates "the distance a robot moves from the current time to the time when the robot starts its stopping action". "S _s " indicates "the distance a robot moves from the time when the robot starts its stopping action to the time when the robot stops moving". "c" indicates "depth penetration factor", for example, "the distance from a detection position such as the center (center of gravity) of a person to a part of the person such as an arm". "Z _d " indicates "the measurement error of the sensor that detects the human body", and "Z _r " indicates "the error of the control position in controlling the robot".

If the configuration of the robot interference determination device is rearranged in accordance with the safe distance Sp, the "estimated position of the object" corresponds to "S _h ," the "estimated position of the movable part" corresponds to "S _r +S _s ," and the "predetermined value" corresponds to "c + Z _d +Z _r ."

In a robot interference determination device according to one aspect of the present invention, the prediction unit may include a learning unit that generates a trained model by learning at least one history of the position, velocity vector, and acceleration vector of the object at each point in time, and calculates the predicted position and the predicted velocity vector of the object using the trained model.

According to the above configuration, the robot interference determination device learns at least one history of the position, velocity vector, and acceleration vector of the object at each point in time to generate a trained model. Then, the robot interference determination device calculates the predicted position and the predicted velocity vector of the object using the generated trained model.

Therefore, the robot interference determination device has the effect of being able to calculate the predicted position and the predicted velocity vector of the object with high accuracy using the learned model generated by learning the history.

In a robot interference determination device according to one aspect of the present invention, the learning unit may generate the learned model by performing learning in which information indicating the change in the position of the object up to the current time is used as input and the acceleration or the subsequent position of the object is output.

According to the above configuration, the robot interference determination device performs learning by taking information indicating the change in the object's position up to the present time as input and outputting the acceleration or subsequent position of the object.

For example, the robot interference determination device learns learning data that takes the position and velocity vector of the object for each time (i.e., at each point in time) as input and the acceleration of the object for each time as output.

The robot interference determination device has the effect of being able to generate a highly accurate trained model for calculating the predicted position and predicted velocity vector of the object by learning training data that receives as input information indicating the change in the object's position up to the current time and outputs the acceleration or subsequent position of the object.

The robot interference determination device according to one aspect of the present invention may further include an identification unit that identifies the object, and the prediction unit may change the method of calculating the predicted position and predicted velocity vector of the object depending on the identification result by the identification unit.

According to the above configuration, the robot interference determination device identifies each of the multiple objects and calculates a predicted position and a predicted velocity vector for each of the multiple objects.

For example, each of the multiple workers has a different height, weight, etc., and each has their own unique movement habits. Therefore, in order to calculate the predicted position and predicted speed vector for each of the multiple workers with high accuracy, it is desirable to learn the historical information of the state and movement of each of the multiple workers and use the learning results.

The robot interference determination device identifies each of the multiple objects and calculates the predicted position and predicted velocity vector of each object using a calculation method appropriate for each object, thereby achieving the effect of being able to calculate the predicted position and predicted velocity vector of each object with high accuracy.

In a robot interference determination device according to one aspect of the present invention, the object state acquisition unit may acquire information indicating changes in the object's position up to the present time based on a history of measurement results obtained by a measurement device that measures the object's two-dimensional or three-dimensional position in a non-contact manner.

According to the above configuration, the robot interference determination device uses the history of the measured positions of the object to obtain information indicating the change in the position of the object up to the present time. The robot interference determination device obtains, for example, the position and velocity vector of the object at the present time as information indicating the change in the position of the object up to the present time.

The robot interference determination device therefore has the effect of being able to calculate and obtain information indicating the change in the object's position up to the present time with high accuracy using the history of the object's measured positions.

In a robot interference determination device according to one aspect of the present invention, the robot state acquisition unit may acquire the predicted position and the predicted velocity vector of the movable part by acquiring control information from a robot control device that controls the operation of the robot.

According to the above configuration, the robot interference determination device obtains the predicted position and the predicted velocity vector of the moving part by obtaining control information that specifies the control of the robot from the robot control device. Therefore, the robot interference determination device has the effect of being able to obtain the predicted position and the predicted velocity vector of the moving part with high accuracy from the control information.

The control information is, for example, an operation program used by the robot control device to execute control over the robot, and the robot interference determination device analyzes the operation program and calculates the predicted position and predicted velocity vector of the moving part.

The robot interference determination device according to one aspect of the present invention may further include an instruction unit that transmits an instruction signal to a robot control device that controls the operation of the robot to instruct the robot to avoid interference when the interference determination unit determines that there is a possibility of contact between the object and the movable part.

According to the above configuration, when the robot interference determination device determines that there is a possibility of contact between the object and the movable part, it transmits an instruction signal to the robot control device that controls the operation of the robot to instruct the robot to avoid interference. Therefore, when the robot interference determination device determines that there is a possibility of contact, it transmits the instruction signal to the robot control device, thereby achieving the effect of being able to avoid contact.

The instruction signal may be a signal (warning signal) that simply warns of the possibility of interference, and the robot control device may determine the specific robot movement to avoid contact. The instruction signal may also be a signal that instructs the robot to move, for example, to stop the robot's movement or to instruct the robot to avoid interference. Furthermore, the instruction signal may be a control command that specifies the robot's movement in detail, for example, a signal that includes information such as the time until the movement is stopped, and the direction, time, and distance of the movement to avoid interference.

In a robot interference determination device according to one aspect of the present invention, when the interference determination unit determines that there is a possibility of contact between the object and the movable part, the instruction unit may transmit the instruction signal to the robot control device, which indicates an instruction to adjust at least one of the moving speed and trajectory of the movable part.

According to the above configuration, when the robot interference determination device determines that there is a possibility of contact between the object and the movable part, it transmits the instruction signal indicating an instruction to adjust at least one of the moving speed and trajectory of the movable part to the robot control device. Therefore, when the robot interference determination device determines that there is a possibility of contact, it adjusts at least one of the moving speed and trajectory of the movable part, thereby achieving the effect of being able to avoid contact.

In a robot interference determination device according to one aspect of the present invention, if the interference determination unit determines that there is no possibility of contact between the object and the movable part after the adjustment instruction is given by the instruction unit, the instruction unit may transmit the instruction signal to the robot control device indicating an instruction to return at least one of the moving speed and trajectory of the movable part to the state before the adjustment instruction was given.

According to the above configuration, when there is no longer any possibility of contact between the object and the movable part after the adjustment instruction, the robot interference determination device instructs the robot control device to return at least one of the moving speed and trajectory of the movable part to the state before the adjustment. Therefore, the robot interference determination device has the effect of being able to return at least one of the moving speed and trajectory of the movable part to the state before the adjustment when there is no longer any possibility of contact between the object and the movable part after the adjustment instruction.

In a robot interference determination device according to one aspect of the present invention, when the interference determination unit determines that a possibility assessment value indicating the possibility of contact between the object and the movable part is equal to or greater than a first threshold, the instruction unit may transmit to the robot control device the instruction signal indicating an instruction to adjust at least one of the moving speed and trajectory of the movable part, and when the interference determination unit determines that the possibility assessment value is equal to or greater than a second threshold higher than the first threshold, the instruction unit may transmit to the robot control device the instruction signal indicating an instruction to stop the movable part.

According to the above configuration, when the possibility evaluation value is equal to or greater than a first threshold, the robot interference determination device instructs adjustment of at least one of the moving speed and trajectory of the movable part, and when the possibility evaluation value is equal to or greater than a second threshold, instructs stopping of the movable part. In other words, the robot interference determination device controls the movement of the movable part according to the possibility of contact between the object and the movable part. When the possibility of contact between the object and the movable part is not very high (i.e., equal to or greater than the first threshold and less than the second threshold), the robot interference determination device adjusts at least one of the moving speed and trajectory of the movable part without stopping the movable part. When the possibility of contact between the object and the movable part is equal to or greater than a second threshold, the robot interference determination device stops the movable part.

The robot interference determination device therefore has the robot perform the necessary and sufficient actions to avoid contact depending on the possibility of contact occurring, thereby achieving the effect of preventing a decrease in production efficiency and space efficiency while avoiding contact.

In order to solve the above problems, a robot interference determination method according to one aspect of the present invention includes an object state acquisition step of acquiring information indicating a change in the position of an object present within a predetermined area up to the present time, a robot state acquisition step of acquiring a predicted position and a predicted velocity vector of a predetermined moving part of the robot after a predetermined time, a prediction step of calculating a predicted position and a predicted velocity vector of the object after the predetermined time based on the information indicating the change in the position of the object up to the present time, and an interference determination step of determining in real time whether or not there is a possibility of contact between the object and the moving part based on the predicted position and the predicted velocity vector of the object and the predicted position and the predicted velocity vector of the moving part.

According to the method, the robot interference determination method acquires information indicating the change in the position of the object up to the present time, and calculates the predicted position and predicted velocity vector of the object using the information indicating the change in the position up to the present time. The information indicating the change in the position of the object up to the present time is, for example, the position of the object at the present time (current position) and the velocity vector of the object at the present time (current velocity vector). In addition, the various pieces of information shown as input in the table of FIG. 15 are each examples of information indicating the change in the position of the object up to the present time.

The robot interference determination method then determines in real time whether or not there is a possibility of contact between the object and the moving part based on the calculated predicted position and predicted velocity vector of the object and the predicted position and predicted velocity vector of the moving part.

The robot interference determination method can avoid interference between the robot and the object without using a safety fence, thereby realizing space savings in production sites, etc.

The robot interference determination method also determines the possibility of interference between the robot and the object at the specified time (i.e., in the future) at the present time, so that even if it is determined that there is a possibility of interference, processing to avoid interference can be executed at the present time with ample time to spare.

Here, if the method for determining the possibility of current interference determines that there is a possibility of current interference, then in order to avoid the current interference, the robot will need to suddenly decelerate and stop, which will inevitably result in a decrease in production efficiency.

In contrast, the robot interference determination method determines the possibility of future interference at the present time, and therefore the robot's movements required to avoid interference can be made smoother and to a minimum, compared to when the possibility of interference at the present time is determined at the present time. In other words, the robot interference determination method can prevent interference between the robot and the object, while suppressing a decrease in production efficiency due to sudden deceleration and stopping of the robot, etc.

The robot interference determination method thus has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the robot and the object, such as a worker, at a production site such as a factory.

In order to solve the above problems, a robot control device according to one aspect of the present invention includes a receiving unit that receives the instruction signal from the robot interference determination device, and a robot control unit that controls the operation of the robot based on the instruction signal.

According to the above configuration, the robot control device receives the instruction signal from the robot interference determination device and controls the operation of the robot based on the received instruction signal. Therefore, the robot control device has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the robot and the object, such as a worker, at a production site such as a factory.

To solve the above problems, a robot control system according to one aspect of the present invention includes the robot interference determination device, the robot control device, and a measurement device that measures the two-dimensional or three-dimensional position of the object in a non-contact manner and transmits the measurement results to the robot interference determination device.

According to the above configuration, the robot control system includes the robot interference determination device, the robot control device, and the measurement device that transmits the measurement results to the robot interference determination device. Therefore, the robot control system has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the robot and the object, such as a worker, at a production site such as a factory.

A human motion prediction device according to one aspect of the present invention includes an object state acquisition unit that acquires information indicating a change in position of a human object, and a prediction unit that predicts a future predicted position, predicted velocity vector, or predicted acceleration vector of the object based on the information indicating the change in position of the object using a trained model, the prediction unit including a learning unit that updates the trained model through online additional learning that does not require parameter adjustment, and the learning unit updates the trained model by learning at least one history of the object's position, velocity vector, and acceleration vector at each point in time.

The above configuration allows the trained model to be updated without parameter adjustment, making it possible to appropriately predict human movements whose movement patterns may change over time.

The learning unit may use DLT (Dynamics Learning Tree) or DBT (Deep Binary Tree) as a learning algorithm.

The learning unit may update the trained model by performing learning using information indicating the change in the position of the object up to a certain point in time as input and outputting the acceleration or the subsequent position of the object.

The prediction unit may predict the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object by inputting information indicating a change in the position of the object over a predetermined period of time into the trained model.

The object may be a worker performing multiple tasks, and the specified period may be set to include not only the period of a first task immediately before the prediction, but also the period of a second task prior to the first task.

With the above configuration, the learning unit can learn the relationships between multiple tasks. This allows the movement of the worker to be predicted appropriately.

The prediction unit may predict the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object by applying a first time-direction weighting filter to information indicating a change in the position of the object during the specified period and using the result as an input to the trained model.

The learning unit may perform learning by applying a second time-direction weighting filter different from the first time-direction weighting filter to information indicating a change in the position of the object during a period including before and after the target time, and the learning unit may predict the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object after the target time by applying the first time-direction weighting filter to information indicating a change in the position of the object during a period before the target time and using the result as an input for the trained model.

According to the above configuration, the time-direction weighting filter used during learning and the time-direction weighting filter used during prediction can be optimized. In other words, it is possible to use a time-direction weighting filter suitable for learning and a time-direction weighting filter suitable for prediction, thereby improving the accuracy of prediction.

The information indicating the change in the position of the object may include the position of a part of the person, a velocity vector, or an acceleration vector.

The information indicating the change in the position of the object may include the positions of multiple parts of the person, a velocity vector, or an acceleration vector.

A human motion prediction method according to one aspect of the present invention includes an object state acquisition step of acquiring information indicating a change in position of a human object, which is an object; a prediction step of predicting a future predicted position, predicted velocity vector, or predicted acceleration vector of the object using a trained model based on the information indicating the change in position of the object; and a learning step of updating the trained model by online additional learning that does not require parameter adjustment, in which the trained model is updated by learning at least one history of the object's position, velocity vector, and acceleration vector at each point in time.

According to one aspect of the present invention, it is possible to avoid a decrease in production efficiency and space efficiency while ensuring the safety of workers at production sites such as factories.

1 is a block diagram showing a configuration of a main part of an interference detection device according to a first embodiment of the present invention. FIG. 2 is a diagram showing an overall outline of a control system including the interference detection device of FIG. 1 . FIG. 3 is a flow chart showing an overview of a process executed in the control system of FIG. 2 . FIG. 4 is a flow chart showing details up to the human trajectory calculation process in the flow chart of FIG. 3. FIG. 4 is a flowchart showing details of a process from a future interference determination process to an arm control process in the flowchart of FIG. 3. 2 is a diagram illustrating a learning process executed by the interference determination device in FIG. 1 and a human trajectory prediction process using a result of the learning process. 2 is a diagram illustrating an example of a future interference detection process executed by the interference detection device of FIG. 1 . FIG. 3 is a diagram illustrating an example of an environment in which the control system of FIG. 2 is applied. 11 is a diagram illustrating an example of a method for calculating the center of gravity of a worker by using the measurement results of a sensor. FIG. FIG. 13 is a diagram illustrating an example of a method for shaping learning data executed before a learning process. FIG. 11 is a diagram illustrating an example of a method for shaping prediction data generated by a learning process. 4 is a diagram showing the accuracy of a human trajectory prediction process executed by the interference determination device of FIG. 1; 13 is a diagram different from FIG. 12 , showing the accuracy of the human trajectory prediction process executed by the interference determination device of FIG. 1 . FIG. 9 is a diagram showing an example of a sensor arrangement different from the example of the sensor arrangement shown in FIG. 8 . 4 is a diagram showing an example of learning data learned by the collision detection device in FIG. 1; FIG. 2 is a block diagram showing a configuration of a main part of a human action prediction device. FIG. 13 is a diagram showing input data and prediction results of a reference example. 11A and 11B are diagrams illustrating input data and prediction results of the human movement prediction device 60 in an operation example. FIG. 13 is a diagram illustrating the relationship between the width of a time window and a prediction error.

Conventional technology that detects the intrusion of a person into a fixed monitoring area and suppresses robot operation tends to reduce production efficiency and space efficiency because it suppresses robot operation when a person intrudes into the monitoring area even when the safety of the person is ensured. Also, conventional technology that limits the torque of a robot tends to reduce production efficiency because it limits the torque of a robot even when the safety of the person is ensured.
[Embodiment 1]
An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to Figs. 1 to 15. Note that the same or corresponding parts in the figures are given the same reference numerals and their description will not be repeated. In this embodiment, for example, an interference detection device 10 will be described as a typical example of a robot interference detection device. To facilitate understanding of the interference detection device 10 according to one aspect of the present invention, first, an overview of a control system 1 including the interference detection device 10 will be described with reference to Fig. 2.

§1. Application example (overall overview of the control system)
Fig. 2 is a diagram showing an overall overview of the control system 1. As illustrated in Fig. 2, the control system 1 includes an interference detection device 10, a controller 20 which is a robot control device, and a sensor 40 (measurement device). In Fig. 2, a sensor 40(1) and a sensor 40(2) are depicted as the sensor 40, but it is not essential that the control system 1 includes a plurality of sensors 40, and the control system 1 may include only one sensor 40. In the following description, when there is no need to distinguish between the sensor 40(1) and the sensor 40(2), they will simply be referred to as "sensor 40".

The control system 1 executes control to avoid inadvertent contact between the robot 30 and the worker 50 (object) in a production site such as a factory where the robot 30 and the worker 50 (object) coexist, as shown in FIG. 2. In particular, the control system 1 executes control to avoid inadvertent contact between the worker 50 and a predetermined part (e.g., a predetermined movable part) of the robot 30 equipped with a multi-joint manipulator, such as the arm of the robot 30. However, in the following description, in order to ensure conciseness of the description, avoiding contact between the worker 50 and the "predetermined part of the robot 30" may be described as avoiding contact between the worker 50 and the robot 30. In other words, in the following description, the term "robot 30" is used not only to refer to the entire robot 30, but also to the "predetermined part of the robot 30" that should be avoided from contacting with the worker 50. Similarly, in the following description, the "velocity vector" and the "acceleration vector" may be abbreviated to "velocity" and "acceleration", respectively.

The sensor 40 mainly measures the two-dimensional or three-dimensional position of the worker 50 (object) in a non-contact manner, and in particular measures the two-dimensional or three-dimensional position of the worker 50 at each time in a non-contact manner, and notifies the collision detection device 10 of the measurement result. The sensor 40 may be, for example, a known LiDAR (Light Detection and Ranging) or RADAR (Radio Detection
The sensor 40 can be realized by using a method such as a Wireless Communication System (WCS) and a Ranging System (WRAN), and is arranged so as to monitor, for example, the entire movable area of the worker 50, that is, the entire working area. The sensor 40 is connected to the interference determination device 10 via an industrial network such as Ethernet (registered trademark).

2, the control system 1 may use multiple sensors 40 to measure the two-dimensional or three-dimensional position of the worker 50. In this case, multiple sensors 40 of different types may be combined, that is, a heterogeneous fusion configuration may be used. For example, in FIG. 2, one of the sensors 40 (1) and 40 (2) may be a two-dimensional sensor, and the other may be a three-dimensional sensor. By combining and using multiple sensors 40 of different types, multi-dimensional measurement becomes possible. In addition, the control system 1 may improve the accuracy of the two-dimensional or three-dimensional position of the worker 50 to be measured by using multiple sensors 40 of the same type. However, as described above, it is not essential that the control system 1 includes multiple sensors 40, and the control system 1 may measure the two-dimensional or three-dimensional position of the worker 50 using one sensor 40.

The controller 20 is a robot control device that controls the operation of the robot 30, and controls the operation of the robot 30 according to data such as a target trajectory and an operation program stored in the arm task 23 described below. The controller 20 also controls the operation of the robot 30 according to an instruction signal from the interference detection device 10. For example, the controller 20 modifies a command value for the robot 30 generated using the target trajectory according to the instruction signal from the interference detection device 10, and causes the robot 30 to perform an operation according to the modified command value. The controller 20 can be realized using, for example, a PLC (Programmable Logic Controller), an IPC (Industrial PC), etc.

The robot 30 is connected to the controller 20 and its operation is controlled by the controller 20, and is equipped with, for example, a multi-joint manipulator in which multiple arms are connected by multiple joints.

The interference detection device 10 predicts the future (in other words, future) "position (two-dimensional or three-dimensional position) and velocity vector" of the worker 50, using the "two-dimensional or three-dimensional position of the worker 50 at each time" acquired from the sensor _40. For example, the interference detection device 10 predicts the "position and velocity vector of the worker 50" at time _t0 + Δt, using the "position and velocity vector of the worker 50" at time _t0 .

Furthermore, the interference detection device 10 acquires information (control information) such as a "target trajectory and motion program" that defines the motion of the robot 30 from the controller 20 (particularly the arm task 23). The interference detection device 10 uses the acquired information to predict (calculate) the future "position (two-dimensional position or three-dimensional position) and velocity vector" of the robot 30 (e.g., the arm of the robot 30). For example, the interference detection device 10 uses the control information of the robot 30 to predict, at time t0, the "position and velocity vector of the robot ₃₀ " at time _t0 + Δt.

The interference detection device 10 calculates a safe distance in the future using a velocity vector of the worker 50 in the future (at time t ₀ + Δt) and a velocity vector of the robot 30. The interference detection device 10 also calculates a separation distance in the future using a position of the worker 50 in the future (at time t ₀ + Δt) and a position of the robot 30 in the future. The interference detection device 10 compares the calculated "safe distance in the future (at time t ₀ + Δt)" with the "separation distance in the future (at time t ₀ + Δt)" to determine the possibility of future interference between the worker 50 and the robot 30, and notifies the controller 20 of the determination result. The concepts of "safety distance" and "separation distance" will be described in detail later with reference to FIG. 6.

Specifically, if the "future separation distance" is greater than the "future safe distance", the interference detection device 10 determines that there is no possibility of interference between the worker 50 and the robot 30 in the future, and notifies the controller 20 of this determination result. Also, if the "future separation distance" is equal to or less than the "future safe distance", the interference detection device 10 determines that there is a possibility of interference between the worker 50 and the robot 30 in the future, and transmits an instruction to avoid interference to the controller 20. The interference detection device 10 may transmit interference state information, such as the future "position and velocity vector" of the worker 50, to the controller 20 along with the instruction to avoid interference.

When the interference detection device 10 determines that there is a possibility of interference, the interference detection device 10 or the controller 20 modifies the planned operation (trajectory plan) of the robot 30 to avoid future interference between the worker 50 and the robot 30.

For example, when the interference detection device 10 determines at the present time (t=t ₀ ) that "the worker 50 and the robot 30 may interfere with each other in the future (t=t ₀ +Δt)", it may transmit an instruction (instruction signal) to the controller 20 to set the "safe distance in the future" to be equal to or less than the "separation distance in the future". That is, the interference detection device 10 may transmit an instruction to the controller 20 at the present time to set the "safe distance in the future" to be equal to or less than the "separation distance in the future". The interference detection device 10 may set the "safe distance in the future" to be equal to or less than the "separation distance in the future" by instructing the controller 20 at the present time to adjust at least one of the "velocity vector and target trajectory" of the robot 30 from the present time to the future.

However, if the interference determination device 10 determines that there is a possibility of interference, it may transmit this determination result and interference status information to the controller 20, and allow the controller 20 to determine the specific corrections to be made to the trajectory plan to avoid interference.

(Interference detection device)
The interference detection device 10 dynamically (i.e., at any time) calculates the "future safety distance" and the "future separation distance", dynamically determines the possibility of interference in the future (interference detection), and notifies the controller 20 of the determination result. When the interference detection device 10 determines that there is a possibility of interference, it instructs the controller 20 to avoid the interference as appropriate, for example, by correcting at least one of the velocity vector and the trajectory of the robot 30. In other words, the interference detection device 10 performs future interference detection from moment to moment, and instructs the controller 20 to correct at least one of the velocity vector and the trajectory of the robot 30 from moment to moment.

Therefore, the interference determination device 10 can avoid interference between the robot 30 and the worker 50 without using a safety fence, thereby realizing space saving in production sites, etc. Furthermore, the interference determination device 10 adjusts at least one of the speed vector and the target trajectory of the robot 30 from the present to the future based on the determination of the possibility of interference in the future, so that a smooth interference avoidance operation by the robot 30 can be planned. In other words, since the operation of the robot 30 is planned using prediction information about the future, it is possible to suppress the occurrence of sudden deceleration, unnecessary deceleration, unnecessary stopping, etc. of the robot 30, and it is possible to generate a trajectory (i.e., an operation plan) with a high success rate. "Low success rate" means, for example, that it is necessary to remake the operation plan (trajectory) of the robot 30 many times in order to avoid interference between the robot 30 and the worker 50. By using the interference determination device 10, a user can realize space saving in production sites, etc. and improvement of production efficiency.

That is, the interference detection device 10 (robot interference detection device) includes a human state information generation unit 112 (object state acquisition unit) that acquires information indicating a change in the position of a worker 50 (object) present within a predetermined area up to the present time (t ₀ ) (for example, a current position x _h (t ₀ ) and a current speed vector v _h (t ₀ ) at the present time), a robot state acquisition unit 121 that acquires a predicted position x _r (t ₀ + Δt) and a predicted speed vector v _r (t ₀ + Δt) of the robot 30 (a predetermined movable part of the robot) a predetermined time Δt later, a human trajectory prediction unit 113 (prediction unit) that calculates a predicted position x _h (t 0 + Δt) and a predicted speed vector v _h (t 0 + Δt) of the worker 50 a predetermined time Δt later based on the information indicating the change in the position of the worker 50 up to the present time, and a human trajectory prediction unit 114 (prediction unit) that calculates a predicted position x _h (t ₀ + Δt) and a predicted speed vector v _h ( _{t 0 +} Δt) of the worker 50 a predetermined time Δt later. and a future interference determination unit 122 ( _interference determination unit) that determines in real time whether or not there is a possibility of contact between the worker 50 and the robot 30 based on the predicted position _xr ( _t0 +Δt) and the predicted velocity vector _vr ( _t0 +Δt) of the robot 30.

According to the above configuration, the collision detection device 10 acquires "information indicating changes in the position of the worker 50 up to the present time" and predicts (calculates) a predicted position _xh ( _t0 +Δt) and a predicted speed vector _vh ( _t0 +Δt) of the worker 50 a predetermined time Δt later using this "information indicating changes in the position of the worker 50 up to the present time." The "information indicating changes in the position of the worker 50 up to the present time" is, for example, the current position _xh ( _t0 ) and current speed vector _vh ( _t0 ) of the worker 50. In addition, the various pieces of information shown as inputs in the table of FIG. 15 are also examples of information indicating changes in the position of the object up to the present time.

Then, the interference determination device 10 determines in real time whether or not there is a possibility of contact between the worker 50 and the robot 30 based on the calculated predicted position _xh ( _t0 +Δt) and predicted velocity vector _vh ( _t0 +Δt) of the worker 50 and the predicted position _xr ( _t0 +Δt) and predicted velocity vector _vr ( _t0 +Δt) of the robot 30.

The interference detection device 10 can avoid interference between the robot 30 and the worker 50 without using a safety fence, thereby realizing space saving in production sites, etc.

Furthermore, the interference determination device 10 determines the possibility of interference between the robot 30 and the worker 50 at a predetermined time Δt (i.e., in the future) at the current time _t0 . Therefore, even if it is determined that there is a possibility of interference, processing to avoid interference can be executed at the current time _t0 with ample time to spare.

Here, if a device _{that determines the possibility of interference at the current time t0 determines} that there is a possibility of interference at the current time _t0 , then in order to avoid the interference at the current time _t0 , it will be necessary to perform actions such as sudden deceleration and sudden stopping of the robot 30, which will inevitably reduce production efficiency.

In contrast, the interference detection device 10 determines the possibility of future interference at the current time t ₀ , and therefore can make the operation of the robot 30 necessary to avoid interference smoother and to a minimum, compared to a case in which the possibility of interference at the current time t ₀ is determined at the current time t _0. In other words, the interference detection device 10 can prevent a decrease in production efficiency due to sudden deceleration and sudden stopping of the robot 30, while avoiding interference between the robot 30 and the worker 50.

Therefore, the interference determination device 10 has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between a person (object), such as a worker 50, and the robot 30 at a production site such as a factory.

As mentioned above, the object may be a person, such as a worker 50, and more specifically, may be an entire person or part of a person, or may be an object other than a person (such as a mobile robot). If the object is an entire person (worker 50), the position of the object is, for example, the position of the center of gravity of the person. If the object is part of a person or an object other than a person, the position of the object is the position of a part of the person, or the position of the object other than a person.

The calculation of the predicted speed vector _vh ( _t0 +Δt) by the human trajectory prediction unit 113 may utilize the results of learning the history of the object's position _xh (t) and speed vector _vh (t) at each point in time. Also, the calculation of the predicted speed vector _vh ( _t0 +Δt) by the human trajectory prediction unit 113 may utilize simple prediction, that is, assume the continuation of the current speed vector _vh ( _t0 ).

Regarding the interference detection device 10, the robot 30 is equipped with a multi-joint manipulator in which multiple arms are connected at multiple joints. The object for which the interference detection device 10 detects interference with the worker 50 may be the entire robot 30, or a predetermined movable part of the robot 30, for example, one or more predetermined parts of the multi-joint manipulator of the robot 30.

According to the above-described configuration, the interference determination device 10 determines, at a current time t0, the possibility of interference between the robot 30 equipped with a multi-joint manipulator having multiple arms connected at multiple joints and the worker ₅₀ after a predetermined time Δt (i.e., in the future).

Therefore, the interference detection device 10 has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between an object, such as a worker 50, and a robot 30 equipped with a multi-joint manipulator at a production site such as a factory.

In the interference determination device 10, the human trajectory prediction unit 113 includes a learning unit 114 that generates prediction data (trained model) by learning at least one history of the position _xh (t), velocity vector _vh (t), and acceleration vector _ah (t) of the worker 50 at each time point. The human trajectory prediction unit 113 calculates a predicted position _xh ( _t0 +Δt) and a predicted velocity vector _vh ( _t0 +Δt) of the worker 50 based on the prediction data.

According to the above configuration, the collision detection device 10 learns at least one history of the position _xh (t), velocity vector _vh (t), and acceleration vector _ah (t) of the worker 50 at each time point to generate a trained model (prediction data). Then, the collision detection device 10 calculates the predicted position _xh ( _t0 +Δt) and predicted velocity vector _vh ( _t0 +Δt) of the worker 50 using the generated trained model.

Therefore, the interference determination device 10 has the advantage of being able to calculate the predicted position xh(t0+Δt) and predicted velocity vector _vh ( _t0 +Δt) of the worker 50 with high accuracy by using prediction data generated by learning the history related to the state and movement (at least one of the position, velocity vector, and acceleration vector) of _{the worker 50} .

In the interference detection device 10, the learning unit 114 receives as input information indicating the change in the position of the worker 50 up to the present time (for example, at least one of the current position _xh ( _t0 ) and the current velocity vector _vh ( _t0 ) of the worker 50) and generates prediction data by performing learning that outputs the acceleration or the subsequent position _xh ( _t0 + Δt) of the worker 50.

According to the above configuration, the interference detection device 10 performs learning by receiving information indicating the change in the position of the worker 50 up to the present time as input, and outputting the acceleration or the subsequent position x _h (t ₀ + Δt) of the worker 50.

For example, the interference detection device 10 learns learning data in which the position and velocity vector of the worker 50 for each time (i.e., at each point in time) are input, and the acceleration of the worker 50 for each time is output.

The interference determination device 10 receives as input information indicating the change in the position of the worker 50 up to the present time, and learns learning data that outputs the acceleration or subsequent position _xh ( _t0 +Δt) of the worker 50, thereby achieving the effect of being able to generate highly accurate prediction data for calculating the predicted position _xh ( _t0 +Δt) and predicted velocity vector _vh ( _t0 +Δt) of the worker 50.

The interference determination device 10 further includes an identification unit 115 that identifies the worker 50, and the human trajectory prediction unit 113 changes the method of calculating the predicted position and predicted velocity vector of the worker 50 depending on the identification result by the identification unit 115.

According to the above-described configuration, the interference detection device 10 identifies each of the multiple workers 50, and calculates the predicted position _xh ( _t0 +Δt) and predicted velocity vector _vh ( _t0 +Δt) of each of the multiple workers 50 according to each of the multiple workers 50.

Here, for example, each of the multiple workers 50 has a different height, weight, etc., and each has a unique movement habit, etc. Therefore, in order to calculate the predicted position _xh ( _t0 +Δt) and the predicted velocity vector _vh ( _t0 +Δt) for each of the multiple workers 50 with high accuracy, it is desirable to learn the state/movement history information of each of the multiple workers 50 and use the learning results.

The interference detection device 10 identifies each of the multiple workers 50 and calculates the predicted position _xh (t0+Δt) and predicted speed vector _vh ( _t0 +Δt) of each of the multiple workers 50 by a calculation method appropriate for each of them, that is, by using prediction data for each of them. Therefore, the interference detection device 10 has the effect of being able to calculate the predicted position _xh ( _t0 +Δt) and predicted speed vector _vh ( _{t0 +} Δt) of each of the multiple workers 50 with high accuracy by using a learned model (prediction data) for each of them.

In the interference detection device 10, the human status information generation unit 112 acquires information indicating the change in the position of the worker 50 up to the present time (e.g., the current position xh(t0) and the current velocity vector _vh ( _t0 )) based on the history of measurement results from a sensor 40 (measuring device) that measures the two-dimensional or three-dimensional position of the _worker 50 in _a non-contact manner.

According to the above configuration, the collision detection device 10 acquires information indicating changes in the position of the worker 50 up to the present time, using the history of the measured position _xh (t) of the worker 50. The collision detection device 10 acquires, for example, the current position _xh ( _t0 ) and current velocity vector _vh ( _t0 ) of the worker 50 at the present time, as information indicating changes in the position of the worker 50 up to the present time.

Therefore, the interference detection device 10 has the advantage of being able to calculate and acquire information indicating the change in the position of the worker 50 up to the present time with high accuracy using the history of the measured positions x _h (t) of the worker 50.

In the interference determination device 10, a robot state acquisition unit 121 acquires control information from a controller 20 (robot control device) that controls the operation of the robot 30, and thereby acquires a predicted position _xr ( _t0 +Δt) and a predicted velocity vector _vr ( _t0 +Δt) of the robot 30.

According to the above configuration, the interference detection device 10 obtains the predicted position _xr ( _t0 +Δt) and predicted speed vector _vr ( _t0 +Δt) of the robot 30 by obtaining control information that specifies control over the robot 30 from the controller 20. Therefore, the interference detection device 10 has the effect of being able to obtain a highly accurate predicted position _xr ( _t0 +Δt) and predicted speed vector _vr ( _t0 +Δt) for the robot 30 from the control information. The control information is, for example, an operation program or the like used when the controller 20 executes control over the robot 30, and the interference detection device 10 analyzes the operation program or the like to calculate the predicted position and predicted speed vector of the robot 30.

The robot state acquisition unit 121 may also analyze position information of the robot 30 (particularly, the moving parts of the robot 30) and calculate a predicted position and a predicted speed vector of the robot 30 (particularly, the moving parts of the robot 30). For example, the robot state acquisition unit 121 may acquire position information of the robot 30 (particularly, the moving parts of the robot 30) from the controller 20, and calculate a predicted position and a predicted speed vector based on the history of the position information.

Furthermore, the robot state acquisition unit 121 may calculate a predicted position and a predicted speed vector of the robot 30 (particularly, the movable parts of the robot 30) from the control signal transmitted from the controller 20 to the robot 30. For example, the robot state acquisition unit 121 may analyze the contents of the control signal transmitted from the controller 20 to the robot 30 (such as the motor control signal for each joint) to calculate a predicted position and a predicted speed vector of the robot 30 (particularly, the movable parts of the robot 30).

The interference determination device 10 further includes a control instruction adjustment unit 123 (instruction unit) that transmits an instruction signal to the controller 20 that controls the operation of the robot 30 to instruct the robot 30 to avoid interference when the future interference determination unit 122 determines that there is a possibility of contact between the worker 50 and the robot 30.

According to the above configuration, when the interference detection device 10 determines that there is a possibility of contact between the worker 50 and the robot 30, it transmits an instruction signal to the controller 20 that controls the operation of the robot 30 to instruct the controller 20 to avoid interference. Therefore, when the interference detection device 10 determines that there is a possibility of contact, it transmits the instruction signal to the controller 20, thereby achieving the effect of being able to avoid contact.

The instruction signal may be a signal (warning signal) that simply warns of the possibility of interference, and the controller 20 may determine the specific operation of the robot 30 to avoid contact. The instruction signal may also be a signal that instructs the robot 30 to stop operating the robot 30 or to avoid interference, for example. The instruction signal may also be a control command that specifies the operation of the robot 30 in detail, and may also be a signal that includes, for example, the time until the operation is stopped, and the direction, time, and distance of the operation to avoid interference.

In the interference determination device 10, when the future interference determination unit 122 determines that there is a possibility of contact between the worker 50 and the robot 30, the control instruction adjustment unit 123 may transmit to the controller 20 the instruction signal indicating an instruction to adjust at least one of the movement speed and trajectory of the robot 30.

According to the above configuration, when the interference determination device 10 determines that there is a possibility of contact between the worker 50 and the robot 30, it transmits to the controller 20 the instruction signal indicating an instruction to adjust at least one of the movement speed and trajectory of the robot 30. Therefore, when the interference determination device 10 determines that there is a possibility of contact, it adjusts at least one of the movement speed and trajectory of the robot 30, thereby achieving the effect of being able to avoid contact.

The controller 20 (robot control device) includes a receiver 22 that receives the instruction signal from the interference detection device 10, and a robot control unit 21 that controls the operation of the robot 30 based on the instruction signal.

According to the above configuration, the controller 20 receives the instruction signal from the interference determination device 10 and controls the operation of the robot 30 based on the received instruction signal. Therefore, the controller 20 has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between the robot 30 and an object such as a worker 50 at a production site such as a factory.

The control system 1 (robot control system) includes an interference detection device 10, a controller 20, and a sensor 40 (measurement device) that non-contactly measures the two-dimensional or three-dimensional position of a worker 50 and transmits the measurement results to the interference detection device 10.

According to the above configuration, the control system 1 includes an interference detection device 10, a controller 20, and a sensor 40 that transmits measurement results (range data) to the interference detection device 10. Therefore, the control system 1 has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between an object, such as a worker 50, and the robot 30 at a production site such as a factory.

The configuration of the interference detection device 10, etc., which has been outlined above using FIG. 2, will now be described in detail with reference to FIG. 1, and the processing executed by the interference detection device 10, etc., will then be described with reference to FIG. 3 to FIG. 5.

§2. Configuration example (details of the collision detection device)
Fig. 1 is a block diagram showing the main configuration of the interference detection device 10. As shown in Fig. 1, the interference detection device 10 includes a human trajectory processing unit 110 and an arbitration unit 120 as functional blocks other than a storage unit 130.

The human trajectory processing unit 110 performs prediction of the future "position (two-dimensional position or three-dimensional position) and velocity vector" of the worker 50, and notifies the arbitration unit 120 of the predicted future "position and velocity vector" of the worker 50. The human trajectory processing unit 110 includes a sensor information acquisition unit 111, a human state information generation unit 112, a human trajectory prediction unit 113, a learning unit 114, and an identification unit 115.

The arbitration unit 120 uses the future "position (two-dimensional position or three-dimensional position) and velocity vector" of each of the worker 50 and the robot 30 to determine the possibility of future interference between the worker 50 and the robot 30. When the arbitration unit 120 determines that there is a possibility of future interference between the worker 50 and the robot 30, it instructs, for example, the controller 20 to avoid future interference between the worker 50 and the robot 30. The arbitration unit 120 includes a robot state acquisition unit 121, a future interference determination unit 122, and a control instruction adjustment unit 123.

In addition to the above-mentioned functional blocks, the interference detection device 10 may also include a configuration for displaying the determination result, but in order to ensure conciseness of the description, configurations that are not directly related to this embodiment are omitted from the description and block diagram. However, in accordance with the actual situation of the implementation, the interference detection device 10 may also include the omitted configuration. The above-mentioned functional blocks illustrated in FIG. 1 can be realized, for example, by a central processing unit (CPU) or the like reading out a program stored in a storage device (storage unit 130) realized by a read only memory (ROM), a non-volatile random access memory (NVRAM), or the like, into a random access memory (RAM) (not shown) and executing the program. Each functional block in the interference detection device 10 will be described below.

(Details of functional blocks other than the memory unit)
The sensor information acquisition unit 111 acquires data (range data) indicating the "two-dimensional or three-dimensional position of the worker 50 at each time" measured by the sensor 40 from the sensor 40, and outputs the acquired range data to the human state information generation unit 112. When the sensor 40 includes a plurality of sensors 40(1) to (n) (n is an integer of 2 or more), such as the sensor 40(1) and the sensor 40(2) illustrated in FIG. 2, the sensor information acquisition unit 111 integrates the range data acquired from each of the plurality of sensors 40. For example, the sensor information acquisition unit 111 acquires range data from the sensor viewpoint of each of the plurality of sensors 40 from each of the plurality of sensors 40, converts the angle and coordinates of each of the acquired plurality of range data, and integrates the plurality of range data. As an example, the sensor information acquisition unit 111 integrates the range data in the X-axis direction and the range data in the Y-axis direction to acquire position data on the X-Y plane. The sensor information acquisition unit 111 outputs the integrated range data to the human state information generation unit 112 .

The human status information generation unit 112 calculates the "two-dimensional or three-dimensional position of the worker 50 at each time" using the range data acquired from the sensor information acquisition unit 111, and stores the calculated "two-dimensional or three-dimensional position of the worker 50 at each time" in the human status history data 131. The human status information generation unit 112 also calculates the "velocity vector of the worker 50 at each time" from the stored data of the human status history data 131 and the "two-dimensional or three-dimensional position of the worker 50 at each time" calculated using the range data acquired from the sensor information acquisition unit 111. The human status information generation unit 112 stores the calculated "velocity vector of the worker 50 at each time" in the human status history data 131.

That is, the human state information generating unit 112 calculates (acquires) the "position and velocity vector of the worker 50 at each time" using the range data acquired from the sensor 40, and outputs the acquired "position and velocity vector of the worker 50 at each time" to the human trajectory predicting unit 113. In the following description, the "position and velocity vector of the worker 50" is also referred to as the "state of the worker 50", and the history of the "position (and velocity vector) of the worker 50 at each time" is also referred to as the "state history (motion history) of the worker 50".

Here, the range data from the sensor 40 contains, for example, measurement results of objects other than the worker 50 as noise. Therefore, the human status information generating unit 112 removes the noise from the range data to calculate the "position of the worker 50 at each time point."

The human status information generating unit 112 calculates, for example, the "position of the center of gravity of the worker 50 at each time (center of gravity position)" as the "position of the worker 50 at each time (two-dimensional position or three-dimensional position)". The "center of gravity" may be rephrased as the "trunk". In other words, the "position of the worker 50 at each time" that the human status information generating unit 112 outputs to the human trajectory predicting unit 113 and stores in the human status history data 131 is the "center of gravity position of the worker 50 at each time". Similarly, the "velocity vector of the worker 50 at each time" that the human status information generating unit 112 outputs to the human trajectory predicting unit 113 and stores in the human status history data 131 is the "velocity vector of the center of gravity of the worker 50 at each time".

Details of the method for calculating the center of gravity position by the human status information generating unit 112 will be described later using Figure 9, but the human status information generating unit 112 calculates the position of the center of gravity of the worker 50 from the presence position information of the worker 50 indicated by the range measurement data (a point cloud indicating the presence of the worker 50).

The human status information generating unit 112 acquires information (identification information) identifying each of the multiple workers 50 from the identifying unit 115, and stores the acquired identification information in combination with the "position and velocity vectors of the workers 50 at each time" in the human status history data 131. The human status information generating unit 112 stores the "position and velocity vectors at each time" for each of the multiple workers 50 in the human status history data 131. In other words, the human status information generating unit 112 stores the history of the "position and velocity vectors at each time" for each of the multiple workers 50 in the human status history data 131.

The human status information generating unit 112 also generates a "current position and velocity vector" for each of the multiple workers 50 from the identification information of the worker 50 acquired from the identification unit 115 and the "current position and velocity vector of the worker 50." The human status information generating unit 112 notifies the human trajectory prediction unit 113 of the generated "current position and velocity vector" for each of the multiple workers 50.

The human trajectory prediction unit 113 inputs the "current position and speed vector of the worker 50" notified by the human state information generation unit 112 into the learned model (prediction data) generated by the learning unit 114, and predicts the "future position and speed vector of the worker 50." The human trajectory prediction unit 113 notifies the arbitration unit 120 (particularly the future interference determination unit 122) of the predicted "future position and speed vector of the worker 50."

Here, the human trajectory prediction unit 113 predicts the "future position and velocity vector of the center of gravity of the worker 50" using, for example, the "current position and velocity vector of the center of gravity of the worker 50" notified by the human state information generation unit 112.

In other words, the human trajectory prediction unit 113 predicts the future (i.e., future) human trajectory (trajectory indicating the movement of the center of gravity position) of the worker 50 from information indicating the center of gravity position of the worker 50 up to the present, and notifies the arbitration unit 120 (particularly, the future interference determination unit 122) of the predicted future human trajectory.

In particular, the human trajectory prediction unit 113 predicts the "future position and speed vector" of each of the multiple workers 50 from the "current position and speed vector" of each of the multiple workers 50 using the prediction data (trained model) of each of the multiple workers 50. The human trajectory prediction unit 113 notifies the arbitration unit 120 (in particular, the future interference determination unit 122) of the predicted "future position and speed vector" of each of the multiple workers 50.

The human trajectory prediction unit 113 uses the results of learning the state history of the worker 50 by the learning unit 114 to predict the "future position and velocity vector of the center of gravity of the worker 50" from the "current position and velocity vector of the center of gravity of the worker 50".

The learning unit 114 inputs the position (particularly, the center of gravity position) and velocity vector of the worker 50 for each time (particularly, the current time) and learns learning data in which the acceleration of the worker 50 for each time is output.

However, it is not essential that the learning data is a combination of "input: position and velocity vector of worker 50 at each time" and "output: acceleration of worker 50 at each time". Details will be described later using FIG. 15, but the learning data may take at least one of the position and velocity vector of worker 50 at each time (particularly the current time) as input, and output the acceleration or subsequent position of worker 50.

The learning unit 114 generates prediction data (particularly, "pre-filter data for prediction" described later) through learning. The learning unit 114 generates prediction data as a learned model through learning using, for example, a learning algorithm called DBT (Deep Binary Tree). In other words, the learning unit 114 generates prediction data as a learned model. The human trajectory prediction unit 113 uses this prediction data to predict "future position and velocity vector of the center of gravity of the worker 50" from "current position and velocity vector of the center of gravity of the worker 50".

When performing batch learning, for example, the learning unit 114 starts learning after a necessary and sufficient amount of state history (movement history) of the worker 50 has been accumulated. After a necessary and sufficient amount of state history (movement history) of the worker 50 has been accumulated, the learning unit 114 learns the latest learning data generated using the state history (movement history) of the worker 50, which is constantly updated to the latest, and updates the learned model (prediction data) from moment to moment. Since the movements of the worker 50 change from moment to moment, in order to predict with high accuracy the "position and velocity vector of the worker 50" a predetermined time after a certain point in time, it is desirable to use the learned model obtained by learning the latest learning data generated using the latest state history at that certain point in time.

It is not essential that the learning unit 114 performs batch learning, and the learning unit 114 may perform online learning. The learning unit 114 may also switch the learning method from batch learning to online learning. For example, the learning unit 114 may perform batch learning on the state history (operation history) of the operator 50 accumulated during test operation of the control system 1 or accumulated for a predetermined period of time from the start of actual operation. Then, after the start of actual operation, or after "generating a trained model by batch learning after the start of actual operation," the learning unit 114 may update the trained model (prediction data) from moment to moment by online learning.

The learning unit 114 generates learning data for each of the multiple workers 50 from the state history (motion history) of each of the multiple workers 50 stored in the human state history data 131, and learns this "learning data for each of the multiple workers 50." The learning unit 114 learns the learning data for each of the multiple workers 50 and generates prediction data (trained model) for each of the multiple workers 50, that is, generates a trained model for predicting "future position and velocity vectors" for each of the multiple workers 50.

The identification unit 115 acquires information (identification information) that identifies each of the multiple workers 50, and notifies the acquired identification information to the human status information generation unit 112. The identification unit 115 may acquire the identification information, for example, from an ID card carried by each of the multiple workers 50. The identification unit 115 may also acquire the identification information, for example, by performing face recognition or the like on each of the multiple workers 50.

The robot state acquisition unit 121 acquires the "position and velocity vector of the robot 30" in the future (i.e., after a predetermined time), and notifies the future interference determination unit 122 of the acquired "position and velocity vector of the robot 30" in the future. For example, the robot state acquisition unit 121 acquires control information (information such as the "target trajectory and operation program" that specifies the operation of the robot 30) used by the controller 20 when controlling the robot 30 from the controller 20 (particularly the arm task 23). The robot state acquisition unit 121 analyzes the acquired control information to acquire the "position and velocity vector of the robot 30" in the future.

The future interference determination unit 122 determines whether or not there is a possibility of contact between the worker 50 and the robot 30, using the "position and velocity vector of the worker 50" in the future (i.e., after a predetermined time) and the "position and velocity vector of the robot 30" in the future. In particular, the future interference determination unit 122 determines from moment to moment whether or not there is a possibility of contact between the worker 50 and the robot 30, using the "position and velocity vector of the worker 50" in the future and the "position and velocity vector of the robot 30" in the future. The future interference determination unit 122 notifies the control instruction adjustment unit 123 of the determination result.

As described above, the human trajectory prediction unit 113 predicts, at the current time (t= _t0 ), the "position and velocity vector of the worker 50" at a time (t= _t0 +Δt) when a predetermined time (Δt) has passed from the current time, and notifies the future interference determination unit 122 of the prediction result. In addition, the robot state acquisition unit 121 acquires, at the current time (t= _t0 ), the "position and velocity vector of the robot 30" at a time (t= _t0 +Δt) when a predetermined time (Δt) has passed from the current time, and notifies the future interference determination unit 122 of the acquired information.

If the distance between the estimated position of the worker 50 and the estimated position of the robot 30 falls within a predetermined value at any time from time " _t0 + Δt" to the time when the movement of the robot 30 stops, the future interference determination unit 122 determines that there is a possibility of contact between the worker 50 and the robot 30. The method of calculating the estimated positions of the worker 50 and the robot 30 will be described in detail later with reference to FIG.

When the future interference determination unit 122 determines that "the worker 50 and the robot 30 may come into contact," the control instruction adjustment unit 123 transmits an instruction signal to the controller 20 to instruct the avoidance of interference between the worker 50 and the robot 30. The control instruction adjustment unit 123 transmits, for example, "an instruction signal to adjust at least one of the movement speed and trajectory of the robot 30 without stopping the operation of the robot 30" or "an instruction signal to stop the operation of the robot 30."

(Details of the memory unit)
The storage unit 130 is a storage device that stores various data used by the collision detection device 10. The storage unit 130 may non-temporarily store (1) a control program, (2) an OS program, (3) an application program for executing various functions of the collision detection device 10, and (4) various data read when the application program is executed by the collision detection device 10. The above data (1) to (4) are stored in a non-volatile storage device such as a read only memory (ROM), a flash memory, an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM (registered trademark)), or a hard disk drive (HDD). The collision detection device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculations and calculation results during various processes executed by the collision detection device 10, and is configured from a volatile storage device such as a random access memory (RAM). Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical constraints, etc. of the collision detection device 10. The storage unit 130 further stores human state history data 131.

The human condition history data 131 stores "the position and velocity vectors of the worker 50 over time" (history), and in particular stores "the position and velocity vectors of the worker 50 over time" (history).

(Controller details)
The controller 20 is a robot control device that controls the operation of the robot 30, and controls the operation of the robot 30 according to data such as a target trajectory and an operation program stored in an arm task 23 described later. The controller 20 also controls the operation of the robot 30 according to an instruction signal from the interference determination device 10. That is, as illustrated in FIG. 1, the controller 20 includes a robot control unit 21, a receiving unit 22, and an arm task 23. The receiving unit 22 receives an instruction signal from the interference determination device 10 and notifies the robot control unit 21 of the received instruction signal. The arm task 23 stores information such as a "target trajectory and operation program" that specifies the operation of the robot 30. The robot control unit 21 controls the operation of the robot 30 based on the instruction signal notified from the receiving unit 22 and information such as a "target trajectory and operation program" stored in the arm task 23.

§3. Operation Example Fig. 3 is a flow diagram showing an outline of the processing executed in the control system 1. As shown in Fig. 3, in the control system 1, first, the sensor information acquisition unit 111 executes a sensor information integration processing (S110), and then the human presence area calculation processing (S120) is executed by the human state information generation unit 112. When the human trajectory calculation processing (S130) is executed by the human trajectory prediction unit 113, the future interference determination unit 122 executes a future interference determination processing (S140) using the result of the human trajectory calculation processing, etc. As a result of the future interference determination processing by the future interference determination unit 122, when it is determined that there is a possibility that the worker 50 and the robot 30 will interfere in the future, the control instruction adjustment unit 123 executes a control instruction adjustment processing (S150). Then, using the result of the control instruction adjustment processing, etc., the controller 20 controls the robot 30, and executes, for example, arm control (S160), which is a processing for controlling the arm of the robot 30.

Figure 4 is a flow diagram showing details up to the human trajectory calculation process in the flow diagram of Figure 3. As shown in Figure 4, the sensor information integration process (S110) in Figure 3 includes a sensor information acquisition process (S111) which is a process for acquiring from the sensor 40 range measurement data indicating "the two-dimensional or three-dimensional position of the worker 50 at each time point (particularly the current two-dimensional or three-dimensional position)" measured by the sensor 40.

As described above, when the sensor 40 includes multiple sensors (e.g., sensor 40(1) and sensor 40(2)), the sensor information acquisition unit 111 acquires range data of each of the multiple sensors 40 from each of the multiple sensors 40. Then, the sensor information acquisition unit 111 integrates the multiple acquired range data (range data integration process, S112).

As shown in FIG. 4, the human presence area calculation process in FIG. 3 includes a human estimation process (S121), a human center of gravity calculation process (S122), and a human center of gravity buffering process (S123).

First, the human status information generating unit 112 removes noise (such as measurement results for objects other than the worker 50) from the range data acquired from the sensor information acquiring unit 111, and extracts only data indicating the presence of the worker 50 from the range data (human estimation process). The human status information generating unit 112 removes noise from the range data acquired from the sensor information acquiring unit 111 (data indicating the current position of the worker 50) by using, for example, the "time-by-time position of the worker 50" (history of the position) stored in the human status history data 131.

The human state information generating unit 112 calculates the current center of gravity position of the worker 50 from the noise-removed range measurement data, i.e., from the data (point cloud) indicating the presence of the worker 50. For example, the human state information generating unit 112 calculates the current center of gravity position of the worker 50 from the average of the point cloud indicating the presence of the worker 50 (human center of gravity calculation process).

The human status information generating unit 112 stores the calculated current center of gravity position of the worker 50 together with the current time in the human status history data 131 (human center of gravity buffering process). The human status information generating unit 112 also calculates the current speed of the worker 50 from the "time-wise position of the worker 50" (history) already stored in the human status history data 131 and the current position of the worker 50. The human status information generating unit 112 stores the calculated "current speed of the worker 50" together with the "current position (center of gravity position) of the worker 50" and the current time in the human status history data 131.

The human trajectory calculation process in FIG. 3 includes a human trajectory learning process (S131), a human trajectory prediction process using the learning results (S132), and a prediction result output process to the arbitration unit (S133), as shown in FIG. 4.

In the human trajectory learning process, the learning unit 114 inputs the position (particularly, the center of gravity position) and velocity vector of the worker 50 for each time (particularly, the current time) and learns learning data in which the acceleration of the worker 50 for each time is output. Through learning, the learning unit 114 generates prediction data as a learned model.

In the human trajectory prediction process, the human trajectory prediction unit 113 inputs the "current position and speed vector" of the worker 50 into prediction data (trained model) to predict the "future position and speed vector of the worker 50." The human trajectory prediction unit 113 then notifies the arbitration unit 120 of the result of the human trajectory prediction process, i.e., the predicted "future position and speed vector of the worker 50" (S133).

Figure 5 is a flow diagram showing the details of the future interference determination process to the arm control process in the flow diagram of Figure 3. As shown in Figure 5, the future interference determination process of Figure 3 includes a robot trajectory acquisition process (S141) and a future safe distance calculation process (S143), etc.

In the robot trajectory acquisition process (S141), the robot state acquisition unit 121 acquires the "position and velocity vector of the robot 30" in the future, and notifies the future interference determination unit 122 of the acquired "position and velocity vector of the robot 30" in the future. For example, the robot state acquisition unit 121 acquires control information related to the robot 30 from the arm task 23, analyzes the acquired control information, and acquires the "position and velocity vector of the robot 30" in the future (i.e., after a predetermined time).

The future interference determination unit 122 determines whether the result of the human trajectory prediction is buffered, that is, whether the "future position and speed vector of the worker 50" has been notified from the human trajectory prediction unit 113 (S142). If the "future position and speed vector of the worker 50" has not been notified from the human trajectory prediction unit 113 (false in S142), the future interference determination unit 122 cannot execute the future safe distance calculation process, and notifies the controller 20 of this fact via the control instruction adjustment unit 123.

When the "future position and velocity vector of the worker 50" has been notified from the human trajectory prediction unit 113 (true in S142), the future interference determination unit 122 executes a future safe distance calculation process (S143). For example, the future interference determination unit 122 calculates a future safe distance based on the future velocity vectors of the worker 50 and the robot 30. The future interference determination unit 122 also calculates a future separation distance based on the future positions of the worker 50 and the robot 30. The future interference determination unit 122 compares the calculated future safe distance with the future separation distance to determine whether the safety distance will be maintained in the future, that is, whether the future separation distance is greater than the future safe distance (S144).

If the safe distance will be maintained in the future (true in S144), the future interference determination unit 122 determines that there is no need to modify the operation of the robot 30 operating in accordance with the control information, and notifies the controller 20 of this via the control instruction adjustment unit 123. If the safe distance will not be maintained in the future (false in S144), that is, if it is determined that "the worker 50 and the robot 30 may come into contact," the future interference determination unit 122 notifies the control instruction adjustment unit 123 of the determination result that "the worker 50 and the robot 30 may come into contact."

When the future interference determination unit 122 notifies the control instruction adjustment unit 123 of the determination result that "the worker 50 and the robot 30 may come into contact," the control instruction adjustment unit 123 executes a safe trajectory generation process (S151). For example, the control instruction adjustment unit 123 compares the future safe distance with the future separation distance, and transmits to the controller 20 an instruction signal to stop the robot 30 or an instruction signal to adjust at least one of the speed and trajectory of the robot 30.

The controller 20 (particularly, the robot control unit 21) executes an arm control process (S160) that controls the operation of the robot 30 (e.g., the arm of the robot 30). When the receiving unit 22 receives a report from the interference detection device 10 that the future safe distance calculation process has not been executed, or that the safe distance will be maintained in the future, the robot control unit 21 controls the robot 30 based on the control information stored in the arm task 23. When the receiving unit 22 receives an instruction signal and a report from the interference detection device 10 that the worker 50 and the robot 30 may come into contact with each other, the robot control unit 21 modifies the control of the robot 30 according to the control information stored in the arm task 23 in accordance with the instruction signal.

The processing executed by the interference detection device 10 that has been described with reference to Figures 3, 4, and 5 (in other words, the robot interference detection method executed by the interference detection device 10) can be summarized as follows. That is, the control method (robot interference detection method) executed by the interference detection device 10 includes an object state acquisition step (human presence area calculation process, S120) of acquiring information indicating a change in the position of the worker 50 (object) present in a predetermined area up to the present time (t ₀ ) (for example, a current position x _h (t ₀ ) and a current speed vector v _h (t ₀ ) at the present time), a robot state acquisition step (robot trajectory acquisition process, S141) of acquiring a predicted position x _r (t ₀ +Δt) and a predicted speed vector v _r (t ₀ +Δt) of the robot 30 (a predetermined movable part of the robot) a predetermined time Δt later, a prediction step (human trajectory prediction process, S132) of calculating a predicted position x _h (t ₀ +Δt) and a predicted speed vector v _h (t ₀ +Δt) of the worker 50 a predetermined time _Δt later based on the information indicating the change in the position of the worker 50 up to the present time, and and an interference determination _step (S144 ₎ of determining _{in real time whether or not there is a possibility of contact between the worker 50 and the robot 30 based on the predicted position xr(t0+Δt) and predicted velocity vector vr ( t0 + Δt} ) of the robot 30.

According to the above method, the robot interference determination method obtains "information indicating changes in the position of the worker 50 up to the present time" and predicts (calculates) a predicted position _xh ( _t0 +Δt) and a predicted velocity vector _vh ( _t0 +Δt) of the worker 50 after a predetermined time Δt using this "information indicating changes in the position of the worker 50 up to the present time." The "information indicating changes in the position of the worker 50 up to the present time" is, for example, the current position _xh ( _t0 ) and current velocity vector _vh ( _t0 ) of the worker 50. Also, the various pieces of information shown as inputs in the table of FIG. 15 are each examples of information indicating changes in the position of the object up to the present time.

The robot interference determination method determines in real time whether or not there is a possibility of contact between the worker 50 and the robot 30 based on the calculated predicted position _xh ( _t0 +Δt) and predicted velocity vector _vh ( _t0 +Δt) of the worker 50 and the predicted position _xr ( _t0 +Δt) and predicted velocity vector _vr ( _t0 +Δt) of the robot 30.

The robot interference determination method can avoid interference between the robot 30 and the worker 50 without using a safety fence, thereby realizing space saving in production sites, etc.

Furthermore, the robot interference determination method determines the possibility of interference between the robot 30 and the worker 50 at a predetermined time Δt (i.e., in the future) at the current time _t0 . Therefore, even if it is determined that there is a possibility of interference, processing to avoid interference can be executed at the current time _t0 with ample time to spare.

Here, if a device _{that determines the possibility of interference at the current time t0 determines} that there is a possibility of interference at the current time _t0 , then in order to avoid the interference at the current time _t0 , it will be necessary to perform actions such as sudden deceleration and sudden stopping of the robot 30, which will inevitably reduce production efficiency.

In contrast, the robot interference determination method determines the possibility of future interference at the current time t ₀ , and therefore can make the operation of the robot 30 necessary to avoid interference smoother and to a minimum, compared to a case in which the possibility of interference at the current time t ₀ is determined at the current time t _0. In other words, the robot interference determination method can prevent interference between the robot 30 and the worker 50, while suppressing a decrease in production efficiency due to sudden deceleration and sudden stopping of the robot 30, etc.

The robot interference determination method thus has the effect of preventing a decrease in production efficiency and space efficiency while avoiding interference between a person (object), such as a worker 50, and the robot 30 at a production site such as a factory.

(Learning process and human trajectory prediction process)
6A and 6B are diagrams for explaining the learning process executed by the interference detection device 10 and the human trajectory prediction process using the results of the learning process. Specifically, (A) of Fig. 6 shows the learning process executed by the interference detection device 10 (particularly, the learning unit 114), and (B) of Fig. 6 shows the human trajectory prediction process executed by the interference detection device 10 (particularly, the human trajectory prediction unit 113).

(About the learning process)
The learning unit 114 calculates "the velocity vector and acceleration vector of the worker 50 at each time" by the following formula, using (the history of) "the position (center of gravity position) of the worker 50 at each time" stored in the human state history data 131. That is, assuming that "x _t " is "the position (center of gravity position) of the worker 50 at time t", "v _t " is "the velocity vector of the worker 50 at time t", and "Δt" is "an infinitesimal time", v _t is expressed as follows:

is defined as:

Furthermore, "a _t " indicating "the acceleration vector of the worker 50 at time t" is expressed as follows:

is defined as:

The learning unit 114 performs learning as shown in Fig. 6A on "the velocity vector _vt and acceleration _vector at of the worker 50 at time t" and "the position (center of gravity position) xt of the worker 50 at time _t " calculated using the above-mentioned formulas 1 and 2. That is, the learning unit 114 learns the output of the acceleration vector at at time t in response to the input of the position _xt at time _t and the velocity vector vt at time _t . The learning unit 114 generates prediction data as a result of learning using, for example, a learning algorithm called DBT.

(About human trajectory prediction processing)
As shown in FIG. 6B, the human trajectory prediction unit 113 obtains "a predicted value apt of the acceleration vector of the worker 50 at time t" using a trained model (prediction data) generated by the learning unit 114 by learning using DBT, for example. That is, the human trajectory prediction unit 113 inputs "the position _xt of the worker 50 at time t" and "the velocity vector _vt of the worker 50 at time _t " into the prediction data (trained model) to obtain "a predicted value apt of the acceleration vector of the worker 50 at time _t " (first prediction process). Then, the human trajectory prediction unit 113 calculates each of "the velocity vector vt+ _Δt of the worker 50 at time t+Δt" and "the position xt+Δt of the worker 50 at time _t+Δt " using the following formula (second prediction process). That is, assuming that "x _t " is the "position (center of gravity) of the worker 50 at time t",""v _t " is the "velocity vector of the worker 50 at time t","_"apt" is the "predicted acceleration vector at time t"," and "Δt" is the "infinite time," v _{t + Δt} is expressed as follows:

It is calculated as:

Furthermore, "x _t+Δt " indicating "the position of the worker 50 at time t+Δt" is expressed as follows:

It is calculated as:

The human trajectory prediction unit 113 repeats the first prediction process and the second prediction process to perform human trajectory prediction, that is, predict the trajectory of change in the position (center of gravity position) of the worker 50.

(Future interference detection process)
7 is a diagram for explaining an example of a future interference detection process executed by the interference detection device 10. The interference detection device 10 executes the future interference detection process using the concept of a "safety distance Sp" defined in 5.5.4.2.3 of ISO/TC15066, for example. The concept of the safety distance Sp will be outlined below.

(Regarding safety distance Sp)
At a certain point in time (time _t0 ), a sensor or the like is used to obtain the "human's velocity vector ( _vh ( _t0 )) and position ( _xh ( _t0 ))" at that point in time, and the "robot's (robot moving part's) velocity vector ( _vr ( _t0 )) and position ( _xr ( _t0 ))" at that point in time. The robot's velocity vector _vr ( _t0 ) and position _xr ( _t0 ) at time _t0 may be calculated using the robot's control information.

The "separation distance (Dr( _t0 ))" between the human and the robot at that certain point in time is calculated from the difference between _xh ( _t0 ) and _xr ( _t0 ). Furthermore, the "shortest distance required for the robot to stop before coming into contact with the human (shortest allowable safe distance ( _Sp ( _t0 )))" at that certain point in time is calculated from _vh ( _t0 ) and vr( _t0 ) using the safety distance calculation formula shown below. Then, Dr( _t0 ) is compared with Sp( _t0 ), and if "Dr( _t0 ) is greater than Sp( _t0 )", it is determined that there will be no interference between the human and the robot, and if "Dr( _t0 ) is equal to or less than Sp( _t0 )", it is determined that there will be interference.

In 5.5.4.2.3 of ISO/TC15066, the safety distance Sp is defined by the following safety distance calculation formula. That is, the safety distance Sp( _t0 ) at a certain time (time _t0 ) is defined as "Sp( _t0 )= _Sh + _Sr + _Ss +c+ _Zd + _Zr ". In the following description, the certain time (time _t0 ) can also be rephrased as "the time (time) when a sensor (e.g., sensor 40) detects a person (e.g., worker 50)". In the following description, it is assumed that a robot (e.g., robot 30) starts to start a stopping operation at "time _t0 + _Tr ", which is _Tr after time _t0 , and that the robot stops at "time _t0 + _Tr + _Ts ", which is _Ts after a further time. Also, the velocity vector of the robot from when the robot starts to stop until the robot stops, that is, the velocity vector of the robot during the deceleration motion, is represented as " _vs (t)". The velocity vector _vs (t) of the robot during the deceleration motion may be calculated using the control information of the robot.

As shown in FIG. 7, in the above-mentioned safe distance calculation formula, "S _h " indicates "the distance that a person will travel from the current time (time t ₀ ) to the time (t ₀ +T _r +T _s ) when the robot stops moving," that is,

It is.

"S _r " indicates "the distance that the robot moves from the current time (time t ₀ ) to the time t ₀ +T _r when the robot starts the stopping operation", that is,

It is.

"S _s " indicates "the distance that the robot moves from time t ₀ +T _r when the robot starts its stopping motion to time t ₀ +T _r +T _s when the robot stops moving", that is,

It is.

In the above-mentioned safe distance calculation formula, "c" indicates the distance from a detection position such as the center of a person (center of gravity) to a part of the person such as an arm, which is called the "depth penetration factor.""Z _d " indicates the "measurement error of the sensor that detects the human body," and "Z _r " indicates the "control position error in robot control."

In the concept of safe distance Sp, a comparison is made between the separation distance Dr( _t0 ) at a certain point in time (time _t0 ) and the shortest safe allowable distance Sp( _t0 ) calculated from the respective velocity vectors ( _vh ( _t0 ) and _vr ( _t0 )) of the human and robot at the certain point in time. If the separation distance Dr( _t0 ) at time _t0 is greater than the shortest safe allowable distance Sp( _t0 ), it is determined that there will be no interference between the human and robot, and if Dr( _t0 ) is equal to or less than Sp( _t0 ), it is determined that there will be interference.

As illustrated in Fig. 7, when the separation distance Dr( _t0 ) is equal to the minimum safe allowable distance Sp( _t0 ), the difference (distance) between "the position (estimated position) of the person at the time ( _t0 + _Tr + _Ts ) when the robot stops moving" and "the position (estimated position) of the robot at the time ( _t0 + _Tr + _Ts ) when the robot stops moving" is equal to " _c +Zd+ _Zr ". As illustrated in Fig. 7, "the estimated position of the person at the time ( _t0 + _Tr + _Ts ) when the robot stops moving" is "the position where the person has moved by _Sh from the position at the current time (time _t0 )". Also, "the estimated position of the robot at the time when the robot stops moving" is "the position where the robot has moved by _Sr + _Ss from the position at the current time (time _t0 )".

Here, by calculating c, _Zd , and _Zr in advance, "c+ _Zd + _Zr " can be treated as a predetermined value. In other words, the concept of the safe distance Sp can be understood as determining that there will be interference between the human and the robot when the distance between the estimated position of the human and the estimated position of the robot at the time when the robot stops moving ( _t0 + _Tr + _Ts ) is equal to or less than a predetermined value (c+ _Zd + _Zr ).

(Application to interference detection device)
The interference detection device 10 calculates the shortest safe allowable distance Sp( _t0 +Δt) in the future using the "position and velocity vector of the worker 50" in the future ( _t0 +Δt) and the "position and velocity vector of the robot 30" in the future. Then, the interference detection device 10 uses the shortest safe allowable distance Sp( _t0 +Δt) in the future ( _t0 +Δt) to determine the possibility of interference between the worker 50 and the robot 30 in the future.

Specifically, first, the human trajectory prediction unit 113 inputs the "position and velocity vector of the worker 50 at the current time (i.e., time _t0 )" into the prediction data generated by the learning unit 114, and predicts the "position and velocity vector of the worker 50 in the future ( _t0 + Δt)". In addition, the robot state acquisition unit 121 analyzes the control information to acquire the "position and velocity vector of the robot 30" in the future ( _t0 + Δt).

Next, the future interference determination unit 122 _calculates the positions of the worker 50 and the robot 30 at the time when the robot stops moving ( _t0 +Δt+ _Tr + _Ts ) from the "position and velocity vectors" of the worker 50 and the robot 30 in the future (t0+Δt). The future interference determination unit 122 calculates the estimated positions of the worker 50 and the robot 30 at the time when the robot stops moving using a safe distance calculation formula from the "position and velocity vectors" of the worker 50 and the robot 30 in the future. The future interference determination unit 122 determines that the human and robot will interfere with each other if the distance between the estimated position of the worker 50 and the estimated position of the robot 30 at the time when the robot stops moving ( _t0 +Δt+ _Tr + _Ts ) is equal to or less than a predetermined value (c+ _Zd + _Zr ).

When the future interference determination unit 122 determines that there will be interference between the human and the robot, the control instruction adjustment unit 123 generates an instruction signal to avoid the interference between the two, for example, to reduce the shortest safe allowable distance Sp ( _t0 +Δt) in the future. In other words, the control instruction adjustment unit 123 generates an instruction signal to adjust at least one of the moving speed and trajectory of the robot 30 to correct the estimated position of the robot 30 at the time when the robot stops moving ( _t0 +Δt+ _Tr + _{Ts )} . The control instruction adjustment unit 123 transmits an instruction signal to the controller 20 to make the distance between the estimated position of the worker 50 and the estimated position of the robot 30 at the time when the robot stops moving ( _t0 + _Δt + _Tr + _Ts ) greater than a predetermined value (c+Zd+Zr).

The interference detection device 10 uses "the position and velocity vectors of the worker 50 and the robot 30 in the future at each time point" to determine the possibility of interference between the worker 50 and the robot 30. Then, when it is determined that there is a possibility of interference, the interference detection device 10 adjusts, for example, at least one of the moving speed and trajectory of the robot 30 to reduce the shortest safe allowable distance Sp( _t0 +Δt) in the future.

The concept of the "safe distance Sp" described with reference to Fig. 7 and the application of this concept to the interference detection device 10 can be summarized as follows: In other words, in the interference detection device 10, the future interference detection unit 122 determines that there is a possibility of contact between the worker 50 and the robot ₃₀ if the distance between the estimated position of the worker 50 and the estimated position of the robot 30 during the time from the current time ( _t0 ) (time t0 + Δt) when a predetermined time Δt has elapsed from the present time (t0) until the robot 30 starts its movement stopping operation and stops moving is within a predetermined value.

According to the above-described configuration, the interference determination device 10 determines that there is a possibility of contact between the worker ₅₀ and the robot 30 when the distance between the estimated position of the worker 50 and the estimated position of the robot 30 falls within a predetermined value at any time from the time point t0 after a predetermined time Δt has elapsed ( _t0 + Δt) until the time point at which the movement of the robot 30 stops.

In other words, the interference determination device 10 makes the above-mentioned determination using not only the distance between the estimated position of the worker 50 and the estimated position of the robot 30 at the time when the movement of the robot 30 stops, but also the distance between the estimated position of the worker 50 and the estimated position of the robot 30 at each point along the way.

Therefore, the interference determination device 10 _{has the effect of being able to make a determination that takes into account the possibility of contact at each point in time from "the point in time t 0 +} Δt, which is the point in time when a predetermined time Δt has elapsed from the current point in time t 0" to "the point in time when the movement of the robot 30 stops."

(Regarding instruction signals)
When the future interference determination unit 122 determines that "the worker 50 and the robot 30 may come into contact with each other," the control instruction adjustment unit 123 transmits an instruction signal to the controller 20 to instruct the controller 20 to avoid interference between the worker 50 and the robot 30. The control instruction adjustment unit 123 may avoid or suppress a decrease in productivity of the robot 30 due to an interference avoidance operation by transmitting an instruction signal such as the following to the controller 20.

(Adjustments to improve productivity)
If the interference detection device 10 determines at time "t= _t0 " that "contact may occur in the future," it adjusts at least one of the speed and trajectory of the robot 30 at time "t= _t0 " in order to avoid contact in the future.

If the interference detection device 10 determines at time "t=t ₀ +t'" that "contact may occur in the future" after adjustment at time "t=t 0", it adjusts at least one of the speed and trajectory of the robot 30 at time "t= _{t 0 +} t'" in order to avoid contact in the future.

If the interference detection device 10 determines at time "t=t ₀ +t'" that "contact will not occur in the future" after the adjustment at time "t=t 0 ", it executes adjustment to improve productivity at time "t= _{t 0} +t'" for at least the speed and trajectory of the robot 30. An adjustment to improve productivity is, for example, an adjustment to return at least one of the speed (acceleration) and trajectory of the robot 30 to the state before the adjustment at time "t=t ₀ ".

In other words, in the interference determination device 10, if the future interference determination unit 122 determines that there is no possibility of contact between the worker 50 and the robot 30 after the adjustment instruction is given by the control instruction adjustment unit 123, the control instruction adjustment unit 123 may transmit to the controller 20 the instruction signal indicating an instruction to return at least one of the movement speed and trajectory of the robot 30 to the state before the adjustment instruction was given.

According to the above configuration, when there is no possibility of contact between the worker 50 and the robot 30 after the adjustment instruction, the interference detection device 10 instructs the controller 20 to return at least one of the movement speed and trajectory of the robot 30 to the state before the adjustment. Therefore, the interference detection device 10 has the effect of being able to return at least one of the movement speed and trajectory of the robot 30 to the state before the adjustment when there is no possibility of contact between the worker 50 and the robot 30 after the adjustment instruction.

(Adjustments according to distance)
Even when it is determined that "contact may occur in the future," if the contact can be avoided by adjusting at least one of the speed and trajectory of the robot 30 rather than stopping the operation of the robot 30, the interference detection device 10 does not stop the operation of the robot 30. The interference detection device 10 adjusts at least one of the speed and trajectory of the robot 30 to avoid contact between the robot 30 and the worker 50.

If it is determined that contact cannot be avoided by simply adjusting at least one of the speed and trajectory of the robot 30, the interference detection device 10 stops the operation of the robot 30. For example, if the distance between the robot 30 and the worker 50 falls below a preset limit distance and it is determined that contact cannot be avoided by simply adjusting at least one of the speed and trajectory of the robot 30, the interference detection device 10 stops the operation of the robot 30.

That is, in the interference determination device 10, when the future interference determination unit 122 determines that the possibility evaluation value indicating the possibility of contact between the worker 50 and the robot 30 is equal to or greater than a first threshold, the control instruction adjustment unit 123 transmits to the controller 20 the instruction signal indicating an instruction to adjust at least one of the movement speed and trajectory of the robot 30. Also, when the future interference determination unit 122 determines that the possibility evaluation value is equal to or greater than a second threshold that is higher than the first threshold, the control instruction adjustment unit 123 transmits to the controller 20 the instruction signal indicating an instruction to stop the robot 30.

According to the above configuration, when the possibility evaluation value is equal to or greater than the first threshold, the interference detection device 10 instructs the adjustment of at least one of the movement speed and trajectory of the robot 30, and when the possibility evaluation value is equal to or greater than the second threshold, instructs the stopping of the robot 30. In other words, the interference detection device 10 controls the movement of the robot 30 according to the possibility of contact between the worker 50 and the robot 30. When the possibility of contact between the worker 50 and the robot 30 is not very high (i.e., equal to or greater than the first threshold and less than the second threshold), the interference detection device 10 adjusts at least one of the movement speed and trajectory of the robot 30 without stopping the robot 30. When the possibility of contact between the worker 50 and the robot 30 is equal to or greater than the second threshold, the interference detection device 10 stops the robot 30.

The interference determination device 10 therefore causes the robot 30 to perform the necessary and sufficient actions to avoid contact depending on the possibility of contact occurring, thereby achieving the effect of preventing a decrease in production efficiency and space efficiency while avoiding contact.

(Regarding the application environment of the control system)
FIG. 8 is a diagram showing an example of an environment to which the control system 1 is applied. As an environment to which the control system 1 is applied, for example, as shown in FIG. 8, a situation can be assumed in which a worker 50 and a robot 30 work in front of a desk and send the results of their work down a conveyer belt in front. In the example shown in FIG. 8, it is assumed that the distance between the passage and the work area is about 1.6 m, a conveyer belt is present, and the height of the desk is 1 m so that the worker 50 can work while standing. In the example shown in FIG. 8, the sensors 40 are arranged on both sides of the worker 50 so as to sandwich the worker 50, thereby making the area of the worker 50 that cannot be detected by the sensors 40 as small as possible.

When placing the sensor 40, the following points were taken into consideration regarding the height at which the sensor 40 is placed. That is, placing the sensor 40 at the height of the head of the worker 50 is undesirable because the measurement waves from the laser or the like of the sensor 40 may be incident on the eyes of the worker 50. Placing the sensor 40 at the height of the waist of the worker 50 is desirable because the results of the core calculation (i.e., the calculation of the center of gravity position) are less likely to be affected by the state of the worker 50. Placing the sensor 40 at the height of the thighs of the worker 50 is undesirable because the results of the core calculation are distorted depending on the extent to which the thighs are spread apart. Placing the sensor 40 at the height of the ankles of the worker 50 is undesirable because the results of the core calculation are distorted depending on the width of the feet of the worker 50 (how far apart the feet are).

(How to calculate the center of gravity)
Fig. 9 is a diagram for explaining an example of a method for calculating the center of gravity of a worker using the measurement results of the sensor 40. In Fig. 9, a star symbol indicates the position of the center of gravity of the worker 50 (center of gravity position), and each point surrounding the star symbol indicates the presence position of the worker 50 (points indicating the presence of the worker 50) indicated by the range measurement data. The human state information generating unit 112 calculates the center of gravity position of the worker 50 indicated by the star symbol from the point cloud indicating the presence of the worker 50 as shown in Fig. 9 using a known center of gravity calculation method.

(Notes on learning process and human trajectory prediction process)
Concerning the human trajectory prediction process and the learning process for generating a learned model used in executing the human trajectory prediction process, the interference determination device 10 specifically executes the processes shown below. The human state information generation unit 112 acquires the "center of gravity position (trunk position) of the worker 50 at each time" and generates learning data for the learning unit 114 to learn from the history of the "center of gravity position (trunk position) of the worker 50 at each time". The learning unit 114 generates "learning unfiltered data (i.e., learning data before execution of filter processing)" from the history of the "center of gravity position (trunk position) of the worker 50 at each time" and shapes the learning unfiltered data to acquire "learning filtered data". Then, the learning unit 114 learns from the learning filtered data and generates "prediction unfiltered data" as a learned model. The human trajectory prediction unit 113 shapes the "prediction unfiltered data" to acquire "prediction filtered data", and predicts "future position and velocity vector of the center of gravity of the worker 50" using the "prediction filtered data". Hereinafter, the shaping process (filter process) for each of the learning data and the prediction data will be described with reference to FIGS. 10 and 11. FIG.

(Regarding the formatting of learning data)
10 is a diagram showing an example of a method for shaping learning data executed before the learning process. The learning unit 114 performs shaping of the learning data according to the following two policies. First, when shaping the learning data, the learning unit 114 generates, as "learning filtered data", a waveform that closely resembles the original data (i.e., learning unfiltered data) as much as possible.

Furthermore, as shown in FIG. 10A, the "actual acceleration (i.e., acceleration before filtering)" becomes almost noise-like data. Therefore, secondly, the learning unit 114 calculates and obtains the "filtered acceleration" shown in FIG. 10B from the "filtered speed" shown in FIG. 10B.

When performing batch learning, for example, the learning unit 114 uses future data as well to apply a weighted moving average filter and generate filtered learning data from the unfiltered learning data. The learning unit 114 uses the generated filtered learning data as learning data and learns from it.

(Regarding data formatting for prediction)
11 is a diagram showing an example of a method for shaping prediction data generated by a learning process. The human trajectory prediction unit 113 performs shaping of prediction data according to the following two policies. First, the human trajectory prediction unit 113 performs prediction in real time, so it uses only past data (prediction data before the present time).

Furthermore, as shown in FIG. 11(A), the "acceleration before filtering" becomes almost noise-like data. Therefore, secondly, the human trajectory prediction unit 113 calculates and obtains the "filtered acceleration" shown in FIG. 11(B) from the "filtered velocity" shown in FIG. 11(B).

The human trajectory prediction unit 113 applies a weighted moving average filter to generate filtered prediction data from unfiltered prediction data, similar to the shaping of learning data by the learning unit 114. The human trajectory prediction unit 113 uses the generated filtered prediction data as a learned model, inputs the "current position and velocity vector of the center of gravity of the worker 50", and predicts the "future position and velocity vector of the center of gravity of the worker 50".

(Accuracy of human trajectory prediction processing)
Since repetitive motions are considered to be the main activity in production sites such as factories, a simple repetitive motion was set as the prediction target, and the prediction accuracy of the human trajectory prediction process executed by the interference determination device 10 was verified through experiments. Specifically, an experiment was conducted on the repetitive motion of "swaying the body from side to side on the spot about 30 times per minute, with the range of movement of the trunk (center of gravity) being about 30 cm."

Figure 12 shows the accuracy of the human trajectory prediction process executed by the interference determination device 10, and in particular shows an excerpt of the results of prediction up to 2 seconds ahead based on prediction data at 0 seconds. As shown in Figure 12, it was confirmed that highly accurate predictions were possible for the human position prediction results up to approximately 0.5 seconds.

Figure 13 is a different figure from Figure 12, showing the accuracy of the human trajectory prediction process executed by the interference determination device 10. Specifically, Figure 13 shows the results when cross-validation was performed as a method for confirming the accuracy of the prediction model (function). That is, first, the data measured by the sensor 40 was divided into a training set and a test set. Next, the explanatory variables of the test set were substituted into the prediction model (function) constructed using the training set to obtain the calculated value of the objective variable. The obtained calculated value was then compared with the measured value of the objective variable of the test set. At that time, the absolute error between the predicted value and the actual measured value was calculated, and the average was calculated for the number of prediction results.

As shown in Figure 12, the verification results shown in Figure 13 also confirmed that highly accurate human position prediction was possible up to about 0.5 seconds.

§4. Modification (Human position detection method)
Fig. 14 is a diagram showing an example of the arrangement of the sensor 40, which is different from the example of the arrangement of the sensor 40 shown in Fig. 8. As shown in Fig. 8, a plurality or a single sensor 40 may be installed near the worker 50, or as shown in Fig. 14(A), a plurality or a single sensor 40 may be installed on the robot 30. Also, as shown in Fig. 14(B), a plurality or a single sensor 40 may be installed on the ceiling of a production site or the like.

(Learning and human trajectory prediction)
So far, an example has been described in which the human trajectory prediction unit 113 predicts "future position and velocity vector of the center of gravity of the worker 50" from "current position and velocity vector of the center of gravity of the worker 50" by using the result of learning of the state history of the worker 50 by the learning unit 114. However, it is not essential for the human trajectory prediction unit 113 to use the result of learning by the learning unit 114 to predict "future position and velocity vector of the center of gravity of the worker 50".

For example, the human trajectory prediction unit 113 may predict the "future position and velocity vector of the worker 50" by simple prediction, that is, by assuming that the "current velocity vector of the worker 50" remains unchanged.

In addition, the learning data used by the learning unit 114 to learn the state history of the worker 50 is not limited to that shown in FIG. 6(A). FIG. 15 shows an example of the learning data that the learning unit 114 learns.

Figure 15 is a diagram showing an example of learning data that the interference detection device 10 learns. In Figure 15, "x" indicates the "position of the center of gravity of the person (worker 50) (center of gravity position)," "v" indicates the "velocity of the center of gravity of the person (worker 50)," and "a" indicates the "acceleration of the center of gravity of the person (worker 50)." Also, "t" indicates the "current time," and "c" indicates the "time the person stopped by time t." In the table shown in Figure 15, each piece of data is one-dimensional or two-dimensional data. In Figure 15, in a pattern that handles data from multiple times, the sampling interval may be not only linear but also nonlinear, and there are also patterns in which the sampling intervals for input and output are different. Each piece of data in the table shown in Figure 15 is, for example, shaped data using a weighted moving average.

As shown in Fig. 15, the learning unit 114 learns at least one history of the position _xh (t), velocity vector _vh (t), and acceleration vector _ah (t) of the worker 50 at each time point. Fig. 15 shows an example of a learning model in which at least one history of the position, velocity vector, and acceleration vector of the worker 50 at each time point (including "at least two of the position, velocity vector, and acceleration vector at the current time point") is input. Similarly, Fig. 15 shows an example of a learning model in which "the history of the position of the worker 50 at each time point" or "the acceleration vector at the current time point" is output.

As described above, the learning unit 114 learns learning data that, for example, takes at least one of the current position and the current velocity vector of the worker 50 as input and outputs the acceleration or the subsequent position of the worker 50.

However, the learning performed by the learning unit 114 is not limited to this. As illustrated in FIG. 15, the learning unit 114 may learn learning data in which "information indicating the change in the position of the worker 50 (target object) up to the present time" is input and "information indicating the change in the position of the worker 50 in the future" is output.

(About the learning algorithm)
In addition, in the examples described so far, the learning unit 114 uses DBT (Deep Binary Tree) as a learning algorithm, but the learning algorithm used by the learning unit 114 is not limited to DBT. The learning unit 114 may execute learning using a neural network or the like, and generate prediction data as a learned model.

[Software implementation example]
The functional blocks of the interference detection device 10 (specifically, the sensor information acquisition unit 111, the human status information generation unit 112, the human trajectory prediction unit 113, the learning unit 114, the identification unit 115, the robot status acquisition unit 121, the future interference detection unit 122, and the control instruction adjustment unit 123) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

In the latter case, the interference determination device 10 includes a CPU that executes the commands of a program, which is software that realizes each function, a ROM (Read Only Memory) or storage device (these are referred to as "recording media") in which the program and various data are recorded so as to be readable by a computer (or CPU), and a RAM (Random Access Memory) in which the program is deployed. The object of the present invention is achieved when the computer (or CPU) reads the program from the recording medium and executes it. The recording medium may be a "non-transient tangible medium," such as a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The program may also be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) capable of transmitting the program. The present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[Embodiment 2]
Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the explanations thereof will not be repeated. In the first embodiment, an example in which interference between the worker and the robot is avoided is described. However, the present invention is not limited to this, and the predicted results of the worker's trajectory may be used for other purposes.

In the first place, in learning algorithms that use neural networks, etc. (e.g., deep learning, etc.), in order to update the trained model through additional learning, it is necessary to manually adjust the learning parameters. Even when a worker (person) repeatedly performs the same task, the worker's movement pattern may change depending on the worker's fatigue, etc. Therefore, in order to make highly accurate predictions, it is necessary to update the trained model. However, when using a learning algorithm that requires manual parameter adjustment, it is difficult to properly update the trained model in real time.

On the other hand, learning algorithms such as DLT (Dynamics Learning Tree) or DBT (Deep Binary Tree) update trained models through online additional learning that does not require parameter adjustment. Therefore, by using DLT or DBT as a learning algorithm and updating trained models in real time, it is possible to appropriately predict worker actions. Here, online additional learning does not simply refer to additional learning via a network, but rather to additional learning in real time using new data (human movements) generated at the site.

[Configuration example]
(Configuration of the human movement prediction device)
16 is a block diagram showing the main components of the human motion prediction device 60. The human motion prediction device 60 includes a human trajectory processing unit 110 and a storage unit 130. The human trajectory processing unit 110 executes prediction of the future "position (two-dimensional position or three-dimensional position) and velocity vector" of the worker 50, and outputs the predicted future "position and velocity vector" of the worker 50 to another external device. The other device may be a server, a controller for controlling a robot, a display device, or the like.

In one aspect of the human movement prediction device 60, the human trajectory prediction unit 113 uses the history of the worker's 50 position over a specified period as input to the learned model (prediction data), updates the learned model, and predicts the "future position of the worker 50."

The human status information generation unit 112 stores the history of the "position by time" of each of the multiple workers 50 in the human status history data 131. The human status information generation unit 112 notifies the human trajectory prediction unit 113 of the generated "current position" of each of the multiple workers 50. The human status information generation unit 112 also notifies the human trajectory prediction unit 113 of the "position of the worker 50 during a specified period including the present" (position history).

The human trajectory prediction unit 113 inputs the "position of the worker 50 during a specified period including the present" notified by the human status information generation unit 112 into the trained model (prediction data) generated by the learning unit 114, and predicts the "future position of the worker 50."

Here, the human trajectory prediction unit 113 predicts the "future position of the center of gravity of the worker 50" using, for example, the "position of the worker 50 during a specified period including the present" (position history) notified by the human status information generation unit 112.

In other words, the human trajectory prediction unit 113 predicts the future (i.e., future) human trajectory (trajectory indicating the movement of the center of gravity position) of the worker 50 from information indicating the center of gravity position of the worker 50 up to the present.

The learning unit 114 learns from learning data in which the position of the worker 50 (particularly the center of gravity position) at each time during a specified period is input, and the position of the worker 50 after the specified period is output. The learning unit 114 generates a trained model using a learning algorithm called DLT or DBT, for example. After starting actual operation (after generating the trained model once), the learning unit 114 updates the trained model using the learning algorithm.

[Example of operation]
4, in the human motion prediction device 60, the sensor information acquisition unit 111 executes a sensor information integration process (S110), the human state information generation unit 112 executes a human presence area calculation process (S120), and the human trajectory prediction unit 113 executes a human trajectory calculation process (S130). However, in the prediction result output process (S133), the prediction result is output to another device instead of the arbitration unit of the first embodiment.

(Updating a trained model)
In the human trajectory learning process (S131, learning step), the learning unit 114 receives the position (position history) of the worker 50 at each time during a predetermined period, and performs learning on learning data in which the position of the worker 50 after the predetermined period is output. The learning unit 114 updates the learned model through learning.

More specifically, the learning unit 114 generates "learning unfiltered data (i.e., learning data before execution of filter processing)" from the history of "the center of gravity position (trunk position) of the worker 50 for each hour" over a predetermined period, and shapes this learning unfiltered data to obtain "learning filtered data". The learning unit 114 applies a learning time direction weighted filter (second time direction weighted filter) to the learning unfiltered data to generate learning filtered data. The learning time direction weighted filter is a weighted moving average filter in the time direction, and is shown below, for example.

_Xt = 0.1xt _-2 + 0.2xt _-1 + 0.4xt ₊ 0.2xt ₊₁ + 0.1xt ₊₂
x indicates a position in the pre-filter data for learning, the subscript t indicates time, and X indicates a position in the filtered data for learning. The learning time-direction weighting filter determines the filtered position Xt at the target time t by weighting and averaging the history of the position x in the period including both before and after the target time _t (from time t-2 to t+2).

The specified period is set in advance so as to include the period of multiple tasks performed by a worker 50 who performs multiple different tasks in sequence. The learning unit 114 updates the trained model by using this training filtered data as input and output for the trained model. Note that when generating a trained model, the learning unit 114 also performs training in a similar manner.

(People trajectory prediction)
In the human trajectory prediction process (S132), the human trajectory prediction unit 113 inputs the position (position history) of the worker 50 at each time during a specified period (including the present time) into the learned model and predicts the "future position of the worker 50."

More specifically, the human trajectory prediction unit 113 generates "prediction unfiltered data (i.e., prediction data before execution of filter processing)" from the history of "the center of gravity position (trunk position) of the worker 50 for each hour" over a predetermined period, and shapes this prediction unfiltered data to obtain "prediction filtered data". The human trajectory prediction unit 113 applies a prediction time direction weighted filter (first time direction weighted filter) to the prediction unfiltered data to generate prediction filtered data. The prediction time direction weighted filter is a weighted moving average filter in the time direction, and is shown, for example, below.

_Xt = 0.05xt _-4 + 0.05xt _-3 + 0.1xt _-2 + 0.2xt _-1 + _0.6xt
x indicates a position in the prediction unfiltered data, the subscript t indicates time, and X indicates a position in the prediction filtered data. The prediction time-direction weighted filter weights and averages the history of the position x in the period before the target time t (from time t-4 to t), and sets the filtered position _Xt at the target time t.

The specified period is set in advance so as to include the periods of multiple tasks performed by a worker 50 who performs multiple different tasks in sequence. The human trajectory prediction unit 113 uses this prediction filtered data as input to the trained model, thereby obtaining the position of the worker 50 in the future from the present time as the output of the trained model. The human trajectory prediction unit 113 can also predict the position of the worker 50 in the further future using the predicted future position of the worker 50. In this way, the human trajectory prediction unit 113 predicts the "future position of the worker 50."

Here, an example is shown in which the learning time direction weighting filter is different from the prediction time direction weighting filter, but the same filter (prediction time direction weighting filter) may be used.

(Reference example prediction results)
FIG. 17 is a diagram showing input data and prediction results of a reference example. The horizontal axis indicates time [s], and the vertical axis indicates the position (x-coordinate, y-coordinate) [m] of the worker 50. Time t1 is the current time, and the position (x-axis input value, y-axis input value) of the worker 50 in a predetermined period P1 before time t1 is used as input data to the trained model to predict the immediate future. The x-axis predicted value and y-axis predicted value after time t1 are prediction results of the position of the worker 50 obtained as an output of the trained model. The x-axis actual measured value and y-axis actual measured value after time t1 indicate the position of the worker 50 that was actually measured later. The position data is data every 0.04 seconds.

In the reference example, the predetermined period P1 is set short enough to include the time of one task A performed by the worker 50 immediately before the prediction. In one task, the worker 50 may not move much. In such a case, the accuracy of the prediction may decrease, and the predicted value (here, the y-axis predicted value) may differ significantly from the actual measured value.

(Prediction results for operation example)
FIG. 18 is a diagram showing input data and prediction results of the human action prediction device 60 in the action example. The horizontal axis indicates time [s], and the vertical axis indicates the position (x-coordinate, y-coordinate) [m] of the worker 50. Time t2 is the current time, and the position (x-axis input value, y-axis input value) of the worker 50 in a predetermined period P2 before time t2 is used as input data to the trained model to predict the immediate future. The x-axis predicted value and y-axis predicted value after time t2 are prediction results of the position of the worker 50 obtained as an output of the trained model. The x-axis actual measured value and y-axis actual measured value after time t2 indicate the position of the worker 50 that was actually measured later. The position data is data every 0.04 seconds.

In the operation example, the predetermined period P2 is set longer than the time of each task. Specifically, the predetermined period P2 is set long enough to include not only the period of one task A performed by the worker 50 immediately before the prediction, but also the period of task B performed before task A. Therefore, the input data to the trained model includes the history of the worker 50's location for multiple tasks A and B. The input data also includes the history of the worker's location during the transition from task A to another task B. In this way, the accuracy of the prediction is improved by using information about the worker 50 in task B immediately before task A for the prediction. The predicted values (x-axis predicted value, y-axis predicted value) in the operation example are in good agreement with the actual measured values.

(Prediction accuracy)
FIG. 19 is a diagram showing the relationship between the width of the time window and the prediction error. The horizontal axis indicates the width of the time window (the length of the predetermined period used as input data) [s]. The vertical axis indicates the prediction error (the difference between the actual value and the predicted value) [m] for the position of the worker 50 at a certain time in the future. It can be seen that if the width of the time window (the length of the predetermined period P2) is small, the prediction accuracy decreases (the prediction error increases). On the other hand, if the width of the time window is relatively long enough to include two different tasks A and B (in this example, if it is 8.96 seconds or more), even if the width of the time window changes slightly, the prediction accuracy does not change much.

[Modification]
In the above, an example has been described in which the position history of the worker 50 is input to the trained model and the future position is output, but the present invention is not limited to this. For example, the position history, velocity vector history, and/or acceleration vector history of the worker 50 may be input to the trained model, and the future position, velocity vector, and/or acceleration vector of the worker 50 may be output from the trained model. The position history, velocity vector history, and/or acceleration vector history of the worker 50 are information indicating changes in the position of the worker 50. In addition, the position history, velocity vector history, and/or acceleration vector history of the worker 50 for a future period may be output from the trained model.

In addition, the position of the worker 50 is not limited to the center of gravity of the worker 50, but may be the position of one or more parts of the worker 50.

The learning unit 114 may also perform learning using a neural network or the like to generate a trained model.

[Software implementation example]
The functional blocks of the human motion prediction device 60 (specifically, the sensor information acquisition unit 111, the human status information generation unit 112, the human trajectory prediction unit 113, the learning unit 114, and the identification unit 115) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

In the latter case, the human motion prediction device 60 includes a CPU that executes the commands of a program, which is software that realizes each function, a ROM (Read Only Memory) or storage device (these are referred to as "recording media") in which the program and various data are recorded so as to be readable by the computer (or CPU), and a RAM (Random Access Memory) in which the program is deployed. The object of the present invention is achieved by the computer (or CPU) reading and executing the program from the recording medium. The recording medium may be a "non-transient tangible medium," such as a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The program may also be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) capable of transmitting the program. The present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments.

1. Control system (robot control system)
10 Interference detection device (robot interference detection device)
20 Controller (robot control device)
21 Robot control unit 22 Receiving unit 30 Robot 40 Sensor (measuring device)
50 Worker (object)
60 Human motion prediction device 112 Human state information generation unit (object state acquisition unit)
113 Person Trajectory Prediction Unit (Prediction Unit)
114 Learning unit 121 Robot state acquisition unit 122 Future interference determination unit (interference determination unit)
115 Identification unit 123 Control instruction adjustment unit (instruction unit)
S120: Person presence area calculation process (object state acquisition step)
S141 Robot trajectory acquisition process (robot state acquisition step)
S132 Human trajectory prediction process (prediction step)
S144 Interference determination step

Claims (10)

an object state acquisition unit that acquires information indicating a change in position of a person, which is an object;
A prediction unit that predicts a future predicted position, a predicted velocity vector, or a predicted acceleration vector of the object based on information indicating a change in the position of the object using a learned model,
The prediction unit includes a learning unit that updates the trained model by online additional learning that does not require parameter adjustment;
A human action prediction device, wherein the learning unit updates the learned model by learning at least one history of the position, velocity vector, and acceleration vector of the object at each point in time. The human movement prediction device according to claim 1, wherein the learning unit uses DLT (Dynamics Learning Tree) or DBT (Deep Binary Tree) as a learning algorithm. The human motion prediction device according to claim 1 or 2, wherein the learning unit updates the learned model by performing learning using information indicating a change in the position of the object up to a certain point in time as input and outputting the acceleration or the subsequent position of the object. The human motion prediction device according to any one of claims 1 to 3, wherein the prediction unit predicts the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object by inputting information indicating a change in the position of the object over a predetermined period of time into the trained model. the object is a worker performing a plurality of tasks,
The human movement prediction device according to claim 4 , wherein the predetermined period is set so as to include not only the period of a first task immediately before prediction, but also the period of a second task prior to the first task. The human motion prediction device according to claim 4 or 5, wherein the prediction unit predicts the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object by applying a first time-direction weighting filter to information indicating a change in the position of the object during the predetermined period and inputting the result into the trained model. the learning unit performs learning by applying a second time direction weighting filter different from the first time direction weighting filter to information indicating a change in position of the object during a period including before and after the target time as an input related to the target time, and
7. The human motion prediction device according to claim 6, wherein the prediction unit predicts the predicted position, the predicted velocity vector, or the predicted acceleration vector of the object after the target time by applying the first time direction weighting filter to information indicating a change in the position of the object in a period before the target time and using the result as an input of the trained model. The human motion prediction device according to any one of claims 1 to 7, wherein the information indicating the change in the position of the object includes the position of a part of the person, a velocity vector, or an acceleration vector. The human motion prediction device according to any one of claims 1 to 8, wherein the information indicating the change in the position of the object includes the positions of multiple parts of the person, a velocity vector, or an acceleration vector. an object state acquisition step of acquiring information indicating a change in position of a person, which is an object;
a prediction step of predicting a future predicted position, a predicted velocity vector, or a predicted acceleration vector of the object based on information indicating a change in the position of the object using a learned model;
A human action prediction method comprising: a learning step of updating the trained model by online additional learning that does not require parameter adjustment, the learning step updating the trained model by learning at least one history of the position, velocity vector, and acceleration vector of the object at each point in time.

Images