JP7282016B2 - robot system - Google Patents

Description

The present invention relates to robot systems.

In recent years, the decrease in the working population has become a problem, and there is a demand for resolving labor shortages and improving productivity in the manufacturing industry. As one of the means to solve these problems, utilization of a collaborative robot system in which humans and robots work in cooperation is expected. Improving the work efficiency of this system is an urgent issue.

Patent Document 1 discloses a human position acquisition unit that acquires a human position, which is the position of a person, and a human position acquisition unit that acquires the human position, for the purpose of appropriately presenting a situation of danger accompanying the motion of a robot. a human vision range determination unit that determines the human vision range depending on and an image generation unit configured to generate image data indicating the position and orientation determined by the position and orientation determination unit. there is

WO2011/089885

Collaborative robots are often set to make an emergency stop when they get closer to a worker than a certain distance. Therefore, if the worker approaches the collaborative robot without predicting its movement, and frequent emergency stops occur, work efficiency will be reduced.

Moreover, the emergency stop function provided in the robot is not always perfect. In this case, it is necessary for the operator to predict the behavior of the robot to avoid collision, but it is difficult to predict the behavior of the robot with high accuracy. For example, when the mobile robot approaches the worker from the worker's blind spot, there is a risk that the worker and the mobile robot may collide when they meet each other.

The danger presentation device described in Patent Literature 1 presents a situation of danger associated with the motion of a robot to a person, thereby improving work efficiency and safety. However, Patent Literature 1 does not describe determining whether or not to present a dangerous situation to a person. For this reason, there are cases where workers are presented with situations that do not actually require avoidance, such as collisions. There is a risk that it will be lost.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to extract and present only the hazards necessary for the worker, reduce the number of emergency stops of the robot, and improve work efficiency and safety.

A robot system according to the present invention includes a robot, a sensor, a control device, and an image display device. The control device includes a robot control section and an image display device control section. and a motion planning unit that plans the motion of the robot and creates motion data, and controls the robot based on the motion data. The image display device control unit includes an approach determination unit and an image generation unit, The image generating unit determines whether or not the human and the robot will come close to each other. The image display device displays a mixed reality image based on the mixed reality data, and when the approach determination unit determines that an approach will occur, Display the object on the image display device.

According to the present invention, it is possible to extract and present only the hazards necessary for the worker, reduce the number of emergency stops of the robot, and improve work efficiency and safety.

1 is an overall configuration diagram showing an outline of a robot system of Example 1; FIG. 2 is a block diagram showing functions of the robot system of Example 1. FIG. 4 is a block diagram showing functions of an approach determination unit according to the first embodiment; FIG. 3 is a block diagram showing functions of an image generation unit of Example 1. FIG. 4 is a flow chart showing the steps from planning to execution of the motion of the robot in Example 1. FIG. 4 is a flow chart showing steps from approach determination to display of a three-dimensional object in Example 1. FIG. 5 is a flow chart showing steps of approach determination processing in the first embodiment. 4 is a flow chart showing the steps of a three-dimensional object generation process in Example 1. FIG. 4 is a time chart showing the timing of displaying a three-dimensional object and executing a robot motion in Example 1; FIG. 11 is a block diagram showing the functions of the robot system of Example 2; FIG. 11 is a block diagram showing functions of an approach determination unit of Example 2; 10 is a flow chart showing steps of approach determination processing in the second embodiment. FIG. 11 is a block diagram showing functions of an image generation unit of Example 3; FIG. 11 is a flow chart showing steps of a three-dimensional object generation process according to the third embodiment; FIG. FIG. 12 is a block diagram showing the functions of the robot system of Example 4; FIG. 12 is a flow chart showing processes from operation planning to execution of the robot system in Example 4. FIG. FIG. 11 is a time chart showing the timing of displaying a three-dimensional object and executing a robot action in Example 4; FIG. FIG. 3 is a schematic configuration diagram showing a situation in which an operator moves between a plurality of robots;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 shows the overall configuration of a robot system 100 of Example 1. As shown in FIG.

As shown in this figure, the robot system 100 includes a control device 101 , a robot 102 , a sensor 103 and an image display device 104 .

The control device 101 has functions of receiving information transmitted from the sensor 103 , controlling the robot 102 , transmitting image data to be displayed on the image display device 104 , and the like.

The robot 102 may be fixed or movable, and may be any type of manipulator, arm robot, mobile robot, self-driving vehicle, automatic guided vehicle (AGV), humanoid robot, or the like.

The sensor 103 detects information about objects, people, passages, etc. around the robot 102 . The sensor 103 may be installed on the robot 102, the image display device 104, or the like, or may be arranged at a plurality of locations. The sensor 103 may be a distance sensor that measures the distance to a person, a camera that recognizes images, a sensor that detects wireless communication such as Wi-fi (registered trademark) or Bluetooth (registered trademark), or a sensor that detects a person. and other means for detecting the approach of the robot.

The image display device 104 displays a virtual robot that synthesizes mixed reality. The image display device 104 displays the future motion of the robot 102 as mixed reality. In this figure, it is a head mount display (HMD) worn by the worker 105 . The image display device 104 may be a mobile device such as a smartphone, a terminal installed on an aisle, a wall, or the like, a projector that projects an image on a screen, a wall, a floor, or the like, or other media.

It should be noted that the worker 105 is a term representatively expressing not only an employee of a factory or the like, but also a person who may approach the robot 102 . Therefore, all persons who may approach the robot 102 are targeted.

The control device 101 may be installed separately into a portion that controls the image display device 104 and a portion that controls the robot 102. In such a case, they are connected by wired communication, wireless communication, or the like. . Furthermore, the control device 101 may be incorporated inside the robot 102 , or the control device 101 may be incorporated inside the image display device 104 .

The robot system 100 aims at efficient cooperation with the worker 105, smooth automatic travel, and the like. There is no fence around the robot 102 and the worker 105 can approach the robot 102 . The image display device 104 is used by the worker 105 to confirm the mixed reality.

Next, the functions of the robot system will be explained.

FIG. 2 is a block diagram showing the functions of the robot system.

In the robot system 100 shown in this figure, a robot 102 , a sensor 103 and an image display device 104 are connected to a control device 101 . Here, the connection method may be wired communication or wireless communication.

The control device 101 includes a robot control section 114 and an image display device control section 115 . The robot controller 114 includes a motion planner 116 and a delayer 117 . The image display device control section 115 includes an approach determination section 118 and an image generation section 119 .

The motion planning unit 116 is a part that plans the motion of the robot 102 . This motion may be a motion for the robot 102 to complete the tasks to be completed at once, or may be a motion for partially completing the task. Also, it is possible to change the motion of the robot by interrupting the execution of the once-planned motion. At that time, if necessary, data from the sensor 103 or another sensor (not shown) may be used.

The delay unit 117 has a function of delaying the execution of motion data by the robot 102 by a predetermined time. In this case, the delay unit 117 may delay transmission of the motion data to the robot 102 by a predetermined time, or delay execution by the robot 102 of the motion data transmitted to the robot 102 by a predetermined time. good.

The proximity determination unit 118 calculates the distance between the robot 102 and the worker based on the data on the robot 102 and/or the worker obtained from the sensor, and compares the distance between the robot 102 and the worker with a threshold to determine whether the robot 102 and the worker are approaching each other. This is the part that determines whether or not

The image generator 119 is a part that obtains motion data planned by the motion planner 116 , converts it into an object, and outputs it to the image display device 104 . The image display device 104 displays objects. The objects are preferably three-dimensional objects, but may also be two-dimensional objects.

If the robot 102 is an arm robot, the motion data obtained from the motion planning unit 116 is angle information of each joint for moving to the target position. Data such as orientation is an example.

The robot control unit 114 and the image display device control unit 115 may belong to physically separate hardware. In that case, the problem that the data of the motion data planned by the motion planning unit 116 to be sent to the image generation unit 119 may be limited depending on the communication band of the communication environment. In order to solve this problem, for example, the motion planning unit 116 sends motion data to the image generation unit 119 only when wireless communication reaches between the robot control unit 114 and the image display device control unit 115. good too. This eliminates the need to send all motion data from all robots, and solves the problem that data is limited by the upper limit of the communication capacity of the communication band.

The proximity determination unit 118 receives the data detected by the sensor 103, calculates the distance between the worker and the robot 102 based on the data, and compares it with a threshold to determine the distance between the worker and the robot 102. This is the part that determines whether or not the two are approaching each other. Here, the data depends on the type of sensor 103 . The data of the sensor 103 is, for example, peripheral point cloud data in the case of a rider whose sensor 103 is attached to a robot. In the case of a camera attached to the robot 102, the image is an RGB image or a distance image of the worker. In the case of the image display device 104 or a camera attached to the worker, the image is an RGB image or a distance image of the robot.

The image generation unit 119 uses the approach determination result between the worker and the robot 102 obtained from the proximity determination unit 118 and the motion data of the robot 102 obtained from the motion planning unit 116 to output to the image display device 104 . is the part that generates the three-dimensional object of

Next, a detailed configuration of the approach determination unit will be described.

FIG. 3 is a block diagram showing functions of an approach determination unit.

In this figure, the approach determination section 118 includes a distance calculation section 130 , a threshold comparison section 131 and a threshold storage section 132 .

The distance calculator 130 calculates the distance between the worker and the robot 102 using data from the sensor. As a method of calculation, for example, when the sensor 103 is a camera attached to a worker or an image display device, a distance image of the robot 102 captured by the camera is used to determine which robot is in the distance image by image recognition. 102 is determined, and the distance of the robot 102 from the camera is calculated. Any other method may be used as long as it is a method for calculating the distance between the worker and the robot 102 .

The threshold comparison unit 131 is a part that compares the distance between the worker and the robot 102 calculated by the distance calculation unit 130 with a threshold. The threshold comparison unit 131 determines that the worker and the robot 102 are approaching each other when the distance is smaller than the threshold. The threshold used here is stored in the threshold storage unit 132, and is preferably set and stored in advance before the robot system 100 is used.

Next, a detailed configuration of the image generator 119 will be described.

FIG. 4 is a block diagram showing functions of the image generator.

In this figure, the image generation unit 119 includes a moving image generation unit 141 , a model storage unit 142 and an output coordinate definition unit 143 .

The animation generation unit 141 uses the motion data read from the motion planning unit 116 and the model data of the robot 102 to dynamically represent a three-dimensional object ( animation) is generated. Model data of the robot 102 used here is stored in the model storage unit 142 . It is desirable to set and store a mixed reality model of the robot 102 in advance before using the robot system 100 . Note that the above animation is also called a "moving image model".

For example, if the robot 102 is an arm robot, in order to display motion data, a mixed reality model of all postures of the robot 102 is stored in the model storage unit 142, and the arm robot uses the model storage unit 142 to process the motion data. Generate a 3D object representing the shape.

The output coordinate definition section 143 is a section that defines the location of the 3D object that is visible to the operator when the 3D object generated by the moving image generation section 141 is output to the image display device 104 as mixed reality. For example, if the robot 102 is an arm robot, the definition may be such that the robot 102 and the three-dimensional object are superimposed as images on the image display device 104 . Also, if the robot 102 is an AGV, it may be defined to display a three-dimensional object where the robot 102 actually operates and moves. This allows the operator to intuitively recognize the future location and motion of the robot 102 .

FIG. 5 is a flow chart showing the steps from robot motion planning to execution.

As shown in the figure, first, in step S001, the motion planning unit 116 plans the motion of the robot 102. FIG. When the robot 102 is an arm robot, the angles and angular velocity commands of each joint with respect to the target position of the tip are planned. If the robot 102 is an AGV, it plans movements such as speed and route to reach the target position. If the robot 102 is neither an arm robot nor an AGV, plan the motions for that robot.

Next, in step S<b>002 , the delay unit 117 transmits motion data to the robot 102 after waiting for the time set in the delay unit 117 .

Next, in step S003, the robot 102 is actuated based on the motion data to perform the motion.

The processes from steps S001 to S003 described above are performed after a series of operations of the robot are completed, returning to the start. However, the present invention is not limited to the above steps, and if the surrounding environment is a dynamic environment, the planning may be repeated every 100 msec, for example.

Next, a process from judging the approach between the robot 102 and the worker 105 to displaying a three-dimensional object will be described.

FIG. 6 is a flow chart showing steps from approach determination to display of a three-dimensional object.

As shown in the figure, in step S004, the approach determination unit 118 determines whether or not the worker is approaching the robot 102 (approach determination processing). Details of step S004 will be described later with reference to FIG.

The presence or absence of approach, which is the result of determination in step S004, is as shown in step S005. If there is an approach between the worker and the robot 102, the process proceeds to step S006 (YES in S005). If there is no approach between the worker and the robot 102, the process returns to step S004 and repeats the approach determination process (NO in S005).

In step S<b>006 , the planned motion data is transmitted to image generator 119 .

Next, in step S<b>007 , the image generator 119 generates a three-dimensional object to be output to the image display device 104 . Details of step S007 will be described later with reference to FIG.

Next, in step S<b>008 , the output coordinate definition unit 143 defines the output location (output coordinates) when outputting the three-dimensional object to the image display device 104 . For example, if the robot 102 is an arm robot and the worker 105 wants the three-dimensional object to overlap with the robot 102 when viewed from the image display device 104, a marker recognizable by the image display device 104 is placed on the robot 102. , and a three-dimensional object may be output based on that marker. In this case, the coordinates of the robot 102 are defined in the working space, set and stored in the image display device 104, and the relative positional relationship is determined using the self-position estimation function of the image display device 104. You can take any method.

Next, in step S009, the three-dimensional object generated in step S007 and the output coordinates defined in step S008 are output to image display device 104. FIG.

Next, in step S010, the image display device 104 displays the three-dimensional object so that it appears at the defined coordinates.

The processes from steps S004 to S010 described above are repeated at a constant cycle of, for example, about 100 msec.

Next, an approach determination process between the worker and the robot 102 by the approach determination unit 118 will be described. This is the portion corresponding to step S004 in FIG.

FIG. 7 is a flow chart showing steps of approach determination processing.

As shown in this figure, first, in step S<b>011 , the distance calculator 130 calculates the distance between the worker 105 and the robot 102 based on the data from the sensor 103 . For example, if the sensor 103 is a camera attached to the robot 102 and can receive distance image data, a method of determining the worker 105 in the distance image by an image recognition function and calculating the distance can be adopted. can. Any other method may be used as long as the distance is calculated according to the type of the sensor 103 .

Next, in step S012, the threshold value stored in the threshold storage unit 132 is compared with the distance between the worker and the robot 102 calculated in S011 to confirm whether the distance is smaller than the threshold value. If the distance is smaller than the threshold (YES in step S013), proceed to step S014. If the distance is not smaller than the threshold (NO in step S013), proceed to step S015. The comparison method in this case is that if the distance is less than or equal to the threshold, it is determined as YES in step S013, and if the distance is not less than or equal to the threshold, it is determined as NO in step S013. Any other method may be used as long as it can determine the close relationship with .

When proceeding to step S014, it is determined that the worker 105 and the robot 102 are approaching each other. On the other hand, when proceeding to step S015, it is determined that the worker 105 and the robot 102 are not approaching each other.

Next, a process in which the image generation unit 119 generates a three-dimensional object will be described. This is the part corresponding to S007 in FIG.

FIG. 8 is a flow chart showing steps of a three-dimensional object generation process.

As shown in the figure, first, in step S021, model data of the robot 102 is read from the model storage unit 142. FIG. Here, the model data to be read may be, for example, CAD data or the like, as long as it can be used as a basis for generating a three-dimensional object to be output to the image display device 104 .

Next, in step S022, the motion data planned by the motion planning unit 116 is read.

Next, in step S023, the moving image generating unit 141 generates a three-dimensional object in which the model reproduces the motion data and is configured to appear to move.

FIG. 9 is a time chart showing the timing of displaying a three-dimensional object and executing a robot action. It is a figure in which time flows toward the right.

This figure shows the process from when the three-dimensional object is displayed on the image display device 104 to when the robot 102 performs an action.

First, the image display device 104 starts to display a three-dimensional object configured such that the model data of the robot 102 reproduces the motion data planned by the motion planning unit 116 . The actual motion of the robot 102 is executed with a delay of the time (delay time) delayed by the delay unit 117 from the start time. In this case, the time for approach determination and the time for transmitting motion data are treated as errors and are not counted as delay time.

In this way, since the worker 105 can confirm the motion of the robot 102 as a three-dimensional object ahead of the delay time, the worker 105 can predict the most recent motion of the robot 102 (for example, the motion of several seconds later). It is possible to avoid the emergency stop of the robot 102 due to the approach, and to obtain the effect of improving the work efficiency.

Furthermore, the approach determination unit 118 allows the worker 105 to check only the most recent motion of the approaching robot 102, so that the worker 105 can easily recognize dangers that actually need to be avoided.

(Example of predicting the future positions of the worker and the robot by the proximity determination unit)
In the first embodiment, only the distance between the worker 105 and the robot 102 is used in the proximity determination process.

On the other hand, in the second embodiment, the predicted future motions of the worker 105 and the robot 102 are used to perform approach determination processing with higher accuracy.

The outline of the functional blocks of the robot system is the same as in FIG.

In this embodiment, the internal configuration of the approach determination unit is slightly different from that of the first embodiment. Also, the difference from the first embodiment is that motion data is transmitted from the motion planning unit to the approach determination unit.

FIG. 10 is a block diagram showing functions of the robot system of the second embodiment.

In the robot system 200 shown in this figure, a robot 202 , a sensor 203 and an image display device 204 are connected to a control device 201 . Here, the connection method may be wired communication or wireless communication.

The control device 201 includes a robot control section 214 and an image display device control section 215 . The robot control section 214 includes a motion planning section 216 and a delay section 217 . The image display device control section 215 includes an approach determination section 218 and an image generation section 219 .

In this figure, unlike FIG. 2, motion data is transmitted from the motion planning unit 216 to the approach determination unit 218 as described above.

The approach determination unit 218 receives the motion data of the robot 202 from the motion planning unit 216 and the position data of the worker from the sensor 203 or the like, and predicts the future motions of the robot 202 and the worker to determine whether the robot 202 and the worker are approaching each other. make a judgment.

Next, the configuration of the functional blocks of the approach determination unit 218 will be described.

FIG. 11 is a block diagram showing the functions of the approach determination unit of this embodiment.

As shown in the figure, the approach determination unit 218 includes a future robot position calculation unit 220 , a dangerous area definition unit 221 , a future worker position calculation unit 222 , and a future position overlap determination unit 223 .

The future robot position calculation unit 220 is a part that calculates to which position the robot 202 will move in the future from the motion data of the robot 202 received from the motion planning unit 216 .

The dangerous area defining section 221 is a section that defines a place determined to be dangerous in the sense that the robot 202 will eventually move to that place from the future robot position calculated by the future robot position calculating section 220 . be.

For example, when the robot 202 is an AGV, the defining method may be a method in which everything within a radius of several meters from the future position is defined as a dangerous area, or another method may be used.

The future worker position calculation unit 222 is a part that calculates the position of the worker from the data about the worker received from the sensor 203, and calculates and predicts the future position of the worker. For example, the future position of the worker is calculated by the following method.

When the sensor 203 is a camera and can acquire a distance image, the position data of the worker is calculated by image recognition, and the position data of the worker after several seconds is calculated. Then, for example, on a straight line that can be generated from two position data about the worker, and assuming that the worker is moving after a few seconds, on a straight line that is the same distance as the distance between the two position data The position (position on the extension line) is estimated to be the worker's position after a few seconds. The method is not limited to this, and other methods may be used as long as the method can calculate the future position of the worker.

The future position overlap determination unit 223 determines whether or not the future worker position estimated by the future worker position calculation unit 222 is included in the dangerous area defined by the dangerous area definition unit 221. , and the portion where it is determined that the worker and the robot 202 are approaching each other.

Next, approach determination processing by the approach determination unit 218 will be described.

FIG. 12 is a flow chart showing steps of approach determination processing.

As shown in the figure, first, in step S031, the approach determination unit 218 reads the motion data of the robot planned by the motion planning unit 216 .

Next, in step S032, the future robot position calculator 220 estimates the future position of the robot 202 by the method described above for the set time.

Next, in step S033, the dangerous area definition unit 221 determines the dangerous area by the method described above from the future position of the robot 202 estimated in step S032.

Next, in step S034, the position data of the worker detected by the sensor 203 is read.

Next, in step S035, the future worker position calculator 222 calculates and predicts the future worker position data for the set time by the method described above.

Next, the future position overlap determination unit 223 determines whether or not the predicted position of the worker will enter the dangerous area. Determine that you are close. If not (NO in step S036), the process proceeds to step S038, and it is determined that the robot 202 and the worker are not approaching each other.

In this way, by utilizing the future positions of the robot 202 and the worker to determine the approach, the accuracy of the approach determination is improved. That is, even in a dynamic environment where the robot 202 moves at high speed, only robots that are likely to come into contact with the operator can be confirmed as three-dimensional objects. This prevents the robot 102 operating at high speed from suddenly approaching the worker 105 and causing an emergency stop during the approach determination process, thereby further improving work efficiency.

(Example of generating continuous images in the image generator)
In the first embodiment, a moving image model representing the motion of the robot is displayed as a three-dimensional object. A system that can grasp all movements at once is also conceivable. This makes it possible to grasp all planned movements at a glance, making it easier for workers to grasp complicated movements and movements that take a long time, avoiding emergency stops due to the approach of the operator and the robot, and improving work efficiency. You can get an improvement effect.

FIG. 13 is a block diagram showing the functions of the image generator of this embodiment.

An image generation unit 319 shown in this figure belongs to the same robot system as that of the first or second embodiment, and functional blocks other than the image generation unit 319 are the same as those of the first or second embodiment. .

The image generation unit 319 includes a continuous image generation unit 320 , a model storage unit 321 and an output coordinate definition unit 322 . The difference from the image generation unit 119 ( FIG. 4 ) in the first embodiment is that a continuous image generation unit 320 is provided instead of the moving image generation unit 141 .

The continuous image generation unit 320 is a part that expresses motion data planned by the motion planning unit in a frame-by-frame manner using robot model data stored in the model storage unit 321 . A plurality of robot model data are displayed as three-dimensional objects to represent, for example, a motion from start to finish at once. It is arbitrary how many frames the model is displayed for the length of the movement.

Next, image generation by the continuous image generation unit 320 will be described.

FIG. 14 is a flow chart showing steps of a three-dimensional object generation process.

In this figure, the steps S041 and S042 are the same as in FIG. 8, and the step S043 is a portion for generating a three-dimensional object that displays the motion of the model frame-by-frame and displays them all at once. is different from FIG.

(Example of inserting approval processing before action execution)
In Examples 1 to 3, the delay section provides a delay time to adjust the timing of the robot's motion.

On the other hand, in Example 4, the operator is allowed to judge and approve the future motion of the robot.

FIG. 15 is a block diagram showing the functions of the robot system of this embodiment.

In the robot system 400 shown in this figure, a robot 402 , a sensor 403 and an image display device 404 are connected to a control device 401 . Here, the connection method may be wired communication or wireless communication.

The control device 401 includes a robot control section 414 and an image display device control section 415 . The robot control unit 414 includes a motion planning unit 416 and a motion approval unit 417 . The image display device control section 415 includes an approach determination section 418 and an image generation section 419 .

The robot control unit 414 shown in this figure has an action approval unit 417 instead of the delay unit, unlike in FIG. Then, the image generation unit 419 transmits a three-dimensional object display completion notification to the action approval unit 417 . Upon receiving the display completion notification of the three-dimensional object, the motion approval unit 417 allows the operator to determine whether or not the robot 402 should actually perform the motion. This determination is made using some external input device or the like. When the operator approves the execution of the action, the action approval unit 417 transmits action data to the robot 402 . Alternatively, the approval may be approval of execution of motion data transmitted to the robot 402 in advance. Here, the external input device may be buttons provided on the image display device 404 . Alternatively, a signal (gesture, hand gesture, etc.) to the sensor 403 may be used as the operator's approval. Also, the person who approves may not only be the worker approaching the robot 402 but also a third party such as a supervisor.

FIG. 16 is a flow chart showing steps from operation planning to execution of the robot system in this embodiment.

In this figure, first, in step S051, the motion planning unit 116 plans the motion of the robot 102. FIG. Then, in step S<b>059 , the image generation unit 419 transmits a three-dimensional object display completion notification to the action approval unit 417 .

In step S060, the worker determines whether or not the robot 402 can actually perform the action. If the action approval unit 417 has received approval from the worker (YES in step S060), the process proceeds to step S061. Then, a command is transmitted from the action approval unit 417 to the robot 402 . Thereafter, in step S062, the robot 402 is actuated based on the motion data to perform the motion.

If the operator's approval is not obtained in step S060 (NO in step S060), the robot 402 does not operate.

FIG. 17 is a time chart showing the timing of displaying a three-dimensional object and executing a robot motion in this embodiment. It is a figure in which time flows toward the right.

This figure shows the process from when the three-dimensional object is displayed on the image display device 404 to when the robot 402 performs an action.

In this figure, after the three-dimensional object display is completed, the robot 402 is executed through a motion approval process.

As a result, when a dangerous operation is expected, the execution of the operation of the robot 402 can be stopped in advance to prevent a collision between the worker and the robot, thereby ensuring safety.

In an actual factory or the like, there are situations where a plurality of robots are arranged and each robot performs different tasks and operations. Even in such a case, when a worker (person) passes through a passage provided between multiple robots, it is possible to avoid dangers such as collisions with the robots, and to prevent the worker from feeling unnecessary stress. It is necessary to maintain a situation in which work can proceed smoothly and to reduce the number of emergency stops of the robot.

FIG. 18 is a schematic configuration diagram showing a situation in which an operator moves between a plurality of robots.

In this figure, a plurality of robots 102a, 102b, 102c, 102d are arranged. A worker 105 is wearing an image display device 104 (HMD) and is about to enter between a plurality of robots 102a, 102b, 102c, and 102d.

In such a situation, when the approach determination unit 118 determines that any of the robots 102a, 102b, 102c, and 102d is approaching the worker 105, the worker 105 is provided with accurate information via the HMD. Tell 105. In this case, it is desirable to display the robots 102a, 102b, 102c, and 102d that are approaching the worker 105 on the HMD so that the worker 105 can easily judge them. In addition, it is desirable to switch the contents displayed on the HMD according to the situation that changes from moment to moment.

In this case, any of the configurations and methods shown in Examples 1 to 4 may be applied.

100, 200, 400: robot system, 101, 201, 401: controller, 102, 202, 402: robot, 103, 203, 403: sensor, 104, 204, 404: image display device, 105: operator, 114 , 214, 414: robot control unit, 115, 215, 415: image display device control unit, 116, 216, 416: motion planning unit, 117, 217: delay unit, 118, 218, 418: approach determination unit, 119, 219, 419: image generation unit, 130: distance calculation unit, 131: threshold comparison unit, 132: threshold storage unit, 141: animation generation unit, 142: model storage unit, 143: output coordinate definition unit, 220: future robot position Calculation unit 221: Dangerous area definition unit 222: Future worker position calculation unit 223: Future position overlap determination unit 319: Image generation unit 320: Continuous image generation unit 321: Model storage unit 322: Output coordinates Definition part, 417: action approval part.

Claims (7)

Equipped with a robot, a sensor, a control device, and an image display device,
The control device includes a robot control unit and an image display device control unit,
The robot control unit includes a motion planning unit that plans motions of the robot and creates motion data, and controls the robot based on the motion data,
The image display device control unit includes an approach determination unit and an image generation unit,
The approach determination unit determines whether or not the robot will approach the human using the sensor , and is based on the motion data from the motion planning unit and the worker position data from the sensor. Predicting future motions of the robot and the worker;
When the approach determination unit determines that the approach will occur, the image generation unit generates data of a virtual object representing a future motion of the robot after a predetermined time based on the motion data. synthesizing mixed reality data based on data obtained from a sensor and the data of the virtual object;
The image display device displays the mixed reality image including the virtual object representing the future motion of the robot after a predetermined time based on the mixed reality data,
The robot system displaying the virtual object on the image display device when the approach determining unit determines that the approach will occur. 2. The robot system according to claim 1, wherein said virtual object is a three-dimensional object. 2. The robot system according to claim 1, wherein said virtual object is not displayed on said image display device when said approach determination unit determines that said approach does not occur. 2. The robot system according to claim 1, wherein said robot control unit further includes a delay unit having a function of delaying execution of said motion data by said robot by a predetermined time. 2. The robot system according to claim 1, wherein said robot control section further includes an action approval section for causing said robot to execute said action data after receiving human approval. An image display device used in the robot system according to claim 1. A controller used in the robot system according to claim 1.

Images