JP2018008347A - Robot system and operation region display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot system capable of easily showing an operation region according to actual operation of a robot, and to provide an operation region display method.SOLUTION: A robot system includes: an imaging part which images a robot; a calculation part which acquires a parameter showing a three-dimensional operation region of the robot on the basis of a program controlling operation of the robot, creates three-dimensional shape data of the operation region of the robot using the parameter, superimposes an actual machine image of the robot on the shape data on the basis of an image of a marker for specifying a position of the robot, and generates an augmented reality space; and a display part which displays the augmented reality space.SELECTED DRAWING: Figure 1

Description

  Embodiments according to the present invention relate to a robot system and an operation area display method.

  If the user inadvertently enters the operation area of the industrial robot, there is a risk that the robot will collide with the user. However, conventionally, it has been difficult to show the user an operation region according to the actual operation of the robot.

JP 2014-180707 A

  Provided are a robot system and an operation area display method capable of easily showing an operation area according to an actual operation of a robot to a user.

  The robot system according to the present embodiment acquires a parameter indicating a three-dimensional motion region of the robot based on an imaging unit that images the robot and a program that controls the motion of the robot, and uses the parameters to operate the robot. A calculation unit that generates three-dimensional shape data of the region, and generates an augmented reality space by superimposing the actual robot image and the shape data based on the marker image for specifying the position of the robot; And a display unit for displaying the augmented reality space.

1 is a block diagram showing an example of the configuration of a robot system according to an embodiment. FIG. 3 is a diagram illustrating an example of an appearance of a robot. 1 is a diagram illustrating an example of a configuration of a robot. The flowchart which shows an example of operation | movement (operation area | region display method) of the robot system 100 by this embodiment. The figure which shows a mode that the simulator 41 is executing the robot control program in simulation. FIG. 3 is a conceptual diagram showing three-dimensional shape data R of an operation area of the robot 1. The conceptual diagram which shows augmented reality space. The block diagram which shows an example of a structure of the robot system according to a modification.

  Embodiments according to the present invention will be described below with reference to the drawings. This embodiment does not limit the present invention.

  FIG. 1 is a block diagram showing an example of the configuration of the robot system according to the present embodiment. The robot system 100 includes a robot 1, a terminal 2, and a marker 3. The robot system 100 simulates a program for controlling the operation of the robot 1 using a three-dimensional simulation program, and creates three-dimensional shape data of the operation region of the robot 1. The robot system 100 indicates the operation area of the robot 1 in augmented reality (AR) by superimposing the shape data on the image of the actual machine of the robot 1.

  The robot 1 is an industrial robot that repeatedly executes the same operation, such as a vertical articulated robot, a horizontal articulated robot, and a linear motion robot, and includes a link, a joint, a slider, and the like that form an arm structure. However, the robot 1 is not limited to this, and may be an arbitrary robot. FIG. 2 is a diagram illustrating an example of the appearance of the robot 1. FIG. 3 is a diagram illustrating an example of the configuration of the robot 1. The robot 1 is, for example, an articulated robot having 6 degrees of freedom. One end side of the robot 1 is fixed to the floor surface F, and a hand 2d is provided on the other end side. The position of the tip E of the hand 2d of the robot 1 is indicated with the robot coordinate system C as reference coordinates. In the robot coordinate system C, a direction perpendicular to the floor surface F on which the robot 1 is arranged is a Z direction, and a direction parallel to the floor surface F is an X direction. Further, a direction perpendicular to the X direction and the Z direction (a direction perpendicular to the paper surface) is defined as a Y direction. The origin of the robot coordinate system C is, for example, a point where the robot 1 is fixed with respect to the floor surface F.

  As shown in FIG. 3, the robot 1 has a plurality of links K <b> 1 to K <b> 7 that form an arm structure. The link K1 is fixed to the floor surface F on which the robot 1 is installed. The link K2 is connected to the link K1 so as to be rotatable around a rotation axis A1 (joint J1) substantially perpendicular to the floor surface F. The link K3 is connected to the link K2 so as to be rotatable around a rotation axis A2 (joint J2) substantially perpendicular to the rotation axis A1. The link K4 is connected to the link K3 so as to be rotatable around a rotation axis A3 (joint J3) substantially parallel to the rotation axis A2. The link K5 is connected to the link K4 so as to be rotatable around a rotation axis A4 (joint J4) substantially perpendicular to the rotation axis A3. The link K6 is connected to the link K5 so as to be rotatable around a rotation axis A5 (joint J5) substantially perpendicular to the rotation axis A4. The link K7 is connected to the link K6 so as to be rotatable about a rotation axis A6 (joint J6) substantially perpendicular to the rotation axis A5. Note that “substantially parallel” and “substantially perpendicular” have a broad meaning including not only strict parallel and vertical but also a slight deviation from parallel and vertical.

  Servo motors are provided on the respective rotation axes A1 to A6 (joints J1 to J6), and each servo motor has angle sensors T1 to T6 for detecting the respective rotation positions. Each servo motor is connected to a controller (not shown) of the robot 1 and operates based on a control command of the controller. The controller controls each servo motor by executing a program for controlling the operation of the robot 1 (hereinafter referred to as a robot control program). Thereby, each rotating shaft A1-A6 (joint J1-J6) is controlled to a desired angle, and the hand 2d can be moved to a desired position and a desired angle. Thus, the robot control program is used for actually operating the robot 1.

  The terminal 2 will be described with reference to FIG. 1 again. The terminal 2 includes an input unit 10, a storage unit 20, an imaging unit 30, a calculation unit 40, and a display unit 50. The terminal 2 may be, for example, a tablet terminal, a smartphone, a personal computer, or the like. The terminal 2 may be configured as a device separate from the robot 1 or may be incorporated in the robot 1.

  The input unit 10 includes an interface (for example, a USB (Universal Serial Bus) terminal) that can be connected to an external device such as the robot 1 and / or a user interface (for example, a touch panel, a keyboard) that allows the user 4 to input data. , Mouse, etc.). The input unit 10 may be configured as a touch panel display together with the display unit 50. The input unit 10 receives a robot control program from the robot 1 or an external device (not shown), and stores the robot control program in the storage unit 20.

  The storage unit 20 may include, for example, a RAM (Random Access Memory) or ROM (Read Only Memory) that functions as a main memory, and an HDD (Hard Disk Drive) or SSD (Solid State Drive) that functions as a storage. . The storage unit 20 stores various programs and data for operating the terminal 2, and also stores a robot control program received from the input unit 10, a three-dimensional simulation program and a three-dimensional CAD (Computer-Aided Design) program to be described later. Can do.

  The imaging unit 30 may be a camera having a function of acquiring image data of the robot 1 or the marker 3 and a function of outputting image data to the storage unit 20 or the calculation unit 40. The imaging unit 30 includes an azimuth magnet and a gyro sensor associated with the camera, and can detect the tilt of the camera.

  The calculation unit 40 includes a simulator 41, an operation area generation unit 42, and an augmented reality space generation unit 43. The computing unit 40 may be constituted by, for example, one or a plurality of CPUs (Central Processor Units) and / or a logic LSI (Large Scale Integration).

  The simulator 41 receives the robot control program from the storage unit 20 and executes the robot control program in a simulated manner. For example, the simulator 41 executes a three-dimensional simulation program and executes at least one cycle of the robot control program on the three-dimensional simulation program. The three-dimensional simulation program can simulate the operation of the robot 1 based on the robot control program without actually operating the actual machine of the robot 1. At this time, the simulator 41 periodically acquires a parameter indicating a three-dimensional motion region of the robot 1 by executing the robot control program. The three-dimensional simulation program may be a commercially available one. For example, the 3D simulation program may be, for example, a robot program creation support tool “TSAssist (registered trademark)” manufactured by Toshiba Machine Co., Ltd.

  The parameters indicating the three-dimensional motion region of the robot 1 include coordinate information indicating the position of at least a part of the robot 1, angle information of joints J1 to J6 between the links K1 to K7 constituting the robot 1, and The at least one piece of posture information indicating the posture of the robot 1 is shown. The coordinate information may be, for example, coordinates (x, y, z) indicating the three-dimensional position of the tip E of the hand 2d. Of course, the coordinate information may be coordinates indicating the three-dimensional positions of the other links K1 to K7 or the joints J1 to J7. The angle information of the joints J1 to J6 indicates the angle between two adjacent links. The posture information is used when the posture (positions K1 to K7) of the robot 1 is not uniquely determined only by the coordinate information and the angle information. For example, even if the position of the tip E of the hand 2d and the angles of the joints J1 to J6 are the same, the posture of the robot 1 can be reversed left and right. In this case, the robot 1 can take two postures (left posture or right posture). As described above, when the posture of the robot 1 is not uniquely determined only by the coordinate information and the angle information, the posture information (for example, the left posture or the right posture) is used. The position and posture of the robot 1 are specified by these parameters.

  The period in which the simulator 41 acquires parameters may be arbitrary. For example, the simulator 41 may acquire parameters every 0.1 seconds after executing the robot control program. The simulator 41 sends the parameters sampled by the simulation of the robot control program to the motion area generator 42.

  The motion region generation unit 42 creates three-dimensional shape data of the motion region of the robot 1 using the parameters sampled by the simulator 41. The operation area of the robot 1 is expressed by the above parameters by executing at least one cycle of the robot control program by the simulator 41. Therefore, the motion region generation unit 42 can create three-dimensional shape data of the motion region of the robot 1 by plotting these parameters in time series. For example, the motion region generation unit 42 creates shape data by drawing parameters using a three-dimensional CAD program. The three-dimensional CAD program may be a commercially available one. For example, the three-dimensional CAD program is PTC Inc. “Creo (PTC Inc. (Corporation of Massachusetts, USA) has intellectual property rights such as copyright) (US registered trademark)” manufactured by the company may be used. The shape data of the motion area is stored in the storage unit 20 and sent to the augmented reality space generation unit 43.

  The augmented reality space generation unit 43 acquires an image of the actual machine of the robot 1 from the imaging unit 30, and acquires three-dimensional shape data of the motion region of the robot 1 from the motion region generation unit 42. The augmented reality space generation unit 43 generates an augmented reality space by superimposing the actual image of the robot 1 and the shape data.

  At this time, the augmented reality space generation unit 43 specifies the position, distance, and angle of the robot 1 with respect to the terminal 2 based on the image of the marker 3 acquired from the imaging unit 30. For example, the shape, pattern, and size of the marker 3 are stored in the storage unit 20 in advance. The augmented reality space generation unit 43 compares the information of the marker 3 stored in the storage unit 20 with the shape, pattern, and size of the marker 3 in the actual image captured by the imaging unit 30, and the robot for the terminal 2 1 position, distance, angle, etc. are specified. Thereby, the augmented reality space generation unit 43 can accurately superimpose the shape data indicating the operation area on the image of the actual machine of the robot 1. The augmented reality space generation unit 43 stores the augmented reality space thus generated in the storage unit 20 and sends it to the display unit 50.

  The display unit 50 displays the augmented reality space from the augmented reality space generation unit 43. That is, the display unit 50 superimposes and displays shape data indicating the operation range on the image of the actual machine of the robot 1. At this time, the display unit 50 displays the shape data in a translucent manner so that the operating range of the robot 1 can be easily understood together with the image of the actual machine of the robot 1.

  FIG. 4 is a flowchart showing an example of the operation (operation area display method) of the robot system 100 according to the present embodiment.

  First, the input unit 10 receives a robot control program from the robot 1 or an external device, and stores this robot control program in the storage unit 20 (S10). In addition to the robot control program, in order to obtain a three-dimensional image of the robot 1, parameters unique to the robot 1 such as the contour data of the three-dimensional model of each link and the number of joints are also required. The unique parameters of the robot 1 may be registered in the storage unit 20 in advance, or may be input together with the robot control program in step S10. At this time, the robot 1 itself does not have to execute an operation based on the robot control program.

  Next, the simulator 41 executes a robot control program in a simulated manner, and periodically acquires a parameter indicating a three-dimensional motion region of the robot 1 (S20). For example, FIG. 5 is a diagram illustrating a state in which the simulator 41 executes a robot control program in a simulated manner. The simulated operation of the robot 1 by simulation may be displayed on the display unit 50 as shown in FIG. Note that a three-dimensional image of the robot 1 can be created using the above-mentioned intrinsic parameters.

  Next, the motion region generation unit 42 creates three-dimensional shape data of the motion region of the robot 1 using parameters (S30). For example, FIG. 6 is a conceptual diagram showing three-dimensional shape data R of the operation area of the robot 1. The shape data R may also be displayed on the display unit 50.

  Next, the imaging unit 30 acquires an image of the marker 3 and an image of the actual machine of the robot 1 (S40). At this time, the imaging unit 30 continuously captures the images of the marker 3 and the robot 1 in real time and causes the display unit 50 to display them. The images of the marker 3 and the robot 1 do not need to be stored in the storage unit 20.

  Next, the augmented reality space generation unit 43 specifies the position, distance, and angle of the robot 1 with respect to the terminal 2 based on the image of the marker 3 (S50). At this time, the augmented reality space generation unit 43 may also use information such as the tilt of the camera of the imaging unit 30.

  Next, the augmented reality space generation unit 43 generates an augmented reality space by superimposing the actual image of the robot 1 and the shape data (S60). Thereafter, the display unit 50 displays the augmented reality space (S70). For example, FIG. 7 is a conceptual diagram showing an augmented reality space. At this time, since the terminal 2 knows the actual position, distance, and angle of the robot 1 relative to the terminal 2, the shape data R can be superimposed on the image P of the robot 1. Thereby, the robot system 100 can show the actual operation area of the robot 1 to the user as an augmented reality space.

  The robot system 100 may create and display this augmented reality space while the robot 1 is stopped, and may create and display the augmented reality space while the robot 1 is operating. In addition, during the imaging of the robot 1 in step S40, the robot system 100 obtains parameters, creates shape data, and generates and displays an augmented reality space (steps S20, S30, and S50). S70) may be executed. Thereby, the time required to present the augmented reality space to the user can be shortened, and the waiting time of the user can be shortened. Further, the terminal 2 may allow the user to rewrite the robot control program stored in the storage unit 20, and may execute steps S20 to S70 on the spot using the updated robot control program. For example, FIG. 8 is a block diagram showing an example of the configuration of a robot system according to such a modification. In this case, when the user stores (overwrites) the updated robot control program in the storage unit 20, the terminal 2 causes the display unit 50 to display a button 55 that causes the augmented reality space to be recreated and redisplayed. When the user touches (presses) the button 55, the terminal 2 recreates a new augmented reality space using the updated robot control program and redisplays it on the display unit 50. As a result, the user can check the operation area of the robot 1 in real time while changing the robot control program.

  As described above, the robot system 100 according to the present embodiment acquires the shape data of the three-dimensional motion region of the robot 1 by executing the robot control program by simulation and simulates the image of the actual machine of the robot 1 and the image. Create an augmented reality space that overlays shape data. Thereby, the robot system 100 can display the actual motion region of the robot 1 in the augmented reality space.

  If the movable region of robot 1 (the maximum range in which robot 1 can operate) is displayed in augmented reality space without using an actual robot control program, the region where robot 1 does not actually pass is also movable. Displayed as an area. Therefore, if the movable area of the robot 1 is set as a prohibited area, even an area where the user does not interfere with the robot 1 and does not cause a problem is displayed in the augmented reality space as a prohibited area.

  On the other hand, the robot system 100 according to the present embodiment creates motion area shape data based on a robot control program that actually controls the robot 1. Therefore, the shape data of the motion region is in accordance with the actual motion region of the robot 1 and does not include a region where the robot 1 does not pass even within the movable region. Therefore, the robot system 100 can display an augmented reality space (AR space) in an appropriate range (minimum required range) according to the actual operation area of the robot 1. That is, even if it is a movable area of the robot 1, an area where the robot 1 does not actually pass is not displayed as an entry prohibition area. As a result, the robot system 100 can easily indicate to the user an operation area according to the actual operation of the robot 1. As a result, the user can intuitively recognize the actual operation region of the robot 1.

  At least a part of the operation region display method according to the present embodiment may be configured by hardware or software. When configured by software, a program for realizing at least a part of the functions of the method may be stored in a recording medium such as a flexible disk or a CD-ROM, and read and executed by a computer. The recording medium is not limited to a removable medium such as a magnetic disk or an optical disk, but may be a fixed recording medium such as a hard disk device or a memory. A program that realizes at least a part of the function of the operation area display method may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be distributed in a state where the program is encrypted, modulated or compressed, and stored in a recording medium via a wired line such as the Internet or a wireless line.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 100 ... Robot system, 1 ... Robot, 2 ... Terminal, 3 ... Marker, 10 ... Input part, 20 ... Memory | storage part, 30 ... Imaging part, 40 ... Calculation unit, 50... Display unit

Claims (8)

An imaging unit for imaging the robot;
Acquiring a parameter indicating a three-dimensional motion region of the robot based on a program for controlling the motion of the robot, creating three-dimensional shape data of the motion region of the robot using the parameter; and An arithmetic unit that generates an augmented reality space by superimposing an image of the actual machine of the robot and the shape data based on an image of a marker for specifying the position of the robot;
A robot system comprising a display unit for displaying the augmented reality space.   The parameter indicates at least one information of coordinate information indicating a position of at least a part of the robot, angle information of joints between a plurality of links constituting the robot, and posture information indicating a posture of the robot. Item 2. The robot system according to Item 1.   The robot system according to claim 1, wherein the parameter is periodically acquired when the program is simulated. A storage unit for storing the program;
The said calculating part produces | generates a new augmented reality space using the said program after the update, when the said program in the said memory | storage part is updated. Robot system.   The said calculating part performs acquisition of the said parameter, creation of the said shape data, and the production | generation of the said augmented reality space, while the said imaging part images the said robot, The any one of Claims 1-4 The robot system according to item.   The robot system according to any one of claims 1 to 5, wherein the arithmetic unit acquires parameters by executing the program with a three-dimensional simulation program.   The robot system according to any one of claims 1 to 6, wherein the arithmetic unit creates the shape data by drawing the parameters with a three-dimensional CAD program. Based on a program for controlling the operation of the robot, a parameter indicating the three-dimensional operation region of the robot is acquired by the arithmetic unit,
Create the three-dimensional shape data of the movement area of the robot using the parameters in the calculation unit,
The image of the marker for specifying the position of the robot and the image of the actual machine of the robot are captured by the imaging unit,
Based on the image of the marker, the augmented reality space is generated by the arithmetic unit by superimposing the actual image of the robot and the shape data,
An operation region display method comprising: displaying the augmented reality space on a display unit.

Images