WO2020067257A1

WO2020067257A1 - Control device

Info

Publication number: WO2020067257A1
Application number: PCT/JP2019/037794
Authority: WO
Inventors: 石川　博一
Original assignee: 日本電産株式会社
Priority date: 2018-09-28
Filing date: 2019-09-26
Publication date: 2020-04-02

Abstract

A control device according to the present invention, which controls a robot, is provided with: a structure database that stores structures of a robot to be controlled; an image input unit that inputs an image of the robot; an associating unit that associates prescribed regions in the image of the robot, input by the image input unit, with the structures of the robot, stored in the structure database; a library database that stores action instructions for the robot and instruction inputs corresponding to the action instructions; an instruction input unit that inputs a prescribed region in the image of the robot and an instruction input corresponding to that region; and an action setting unit that sets the action instruction in the library database, corresponding to the instruction input input by the instruction input unit, as the action instruction corresponding to the instruction input.

Description

Control device

The present invention relates to a control device for controlling a robot.

A control device called a programming pendant for instructing an operation of a robot by an image is known. For example, Japanese Patent Laid-Open Publication No. 2007-242054 discloses that a conventional character-based linguistic expression and editing method for a robot language is changed to a graphical expression, thereby facilitating grasping of teaching contents and for beginners. Discloses a technique for easily teaching programming and work programs.

Japanese Unexamined Patent Publication: JP-A-2007-242054

However, in the technique disclosed in Japanese Patent Application Laid-Open Publication No. 2007-242054, the movement path of the robot can be designated by a predetermined graphical representation for a predetermined process such as a welding path. Changes in the expression itself are not taken into account. For this reason, it is necessary to get used to predetermined graphical expressions and operations, and specialty is required.
In view of the above-mentioned problems, an object of the present invention is to provide a control device that allows a user to easily set the operation of a robot.

According to an embodiment of the present invention, there is provided a control device for controlling a robot, comprising: inputting a structure database holding a structure of a robot to be controlled and an image of the robot; An image input unit, a correspondence unit that associates a predetermined area in the image of the robot input by the image input unit with the structure of the robot held in the structure database, and an operation instruction to the robot. A library database that holds an instruction input corresponding to the operation instruction, a predetermined area in the image of the robot and an instruction input unit that inputs an instruction input corresponding to the area, and the library input by the instruction input unit. The operation instruction in the library database corresponding to the instruction input is set as the operation instruction corresponding to the instruction input. Control device is provided, characterized in that it comprises an operation setting unit, the for.

According to the present invention having the above configuration, the user can easily set the operation of the robot.

It is a block diagram showing the example of composition of the robot control system using the control device of an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a configuration of a robot to be controlled. It is a figure showing an example of a DH parameter. 3 is a flowchart illustrating an example of a control process of the robot by the control device illustrated in FIG. 1. It is a figure showing an example of an image of a robot. It is a figure showing the example of the screen displayed when performing correspondence. It is a figure showing the example of the screen displayed when performing correspondence. FIG. 4 is a diagram illustrating an example of data stored in a library DB. It is a figure showing an example of specification of a field and instruction input. It is a figure showing an example of specification of a field and instruction input. It is a figure showing an example of an operation input screen. It is a figure showing an example of a display picture at the time of performing composition of operation. FIG. 10 is a diagram illustrating an example of a surface where an operation input of a synthesized operation is a surface; It is a figure showing other examples of an image of a robot. It is a figure showing other examples of an image of a robot.

Hereinafter, an embodiment for carrying out the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a configuration example of a robot control system using a control device according to an embodiment of the present invention. The robot control system includes a robot 1 to be controlled, a controller 2 for controlling a motor provided in the robot 1, a camera 3 for imaging the robot 1, and the like, and a control device 10 for controlling the entire system. ing.

Hereinafter, a case where an articulated robot having a plurality of joints and arms as shown in FIG. 2 will be described as the robot 1 to be controlled.
The robot 1 includes a plurality of

joints

41a, 41b, 42a, 42b, 43a, 43b, and a hand 44 attached to a tip of the joint 43b. The joint portion 41a is rotatably attached to the base 40. The arm section has a plurality of

arms

45 and 46. The hand unit 44 moves, for example, a member to be processed. FIG. 3 shows an example of DH (Denavit-Hartenberg) parameters used for controlling the robot 1. For example, the DH parameter determines the position of the hand unit 44 at the tip of the robot 1 or specifies the position of the hand unit 44 in accordance with the angles of the joints 41a to 43b and the lengths of the

arms

45 and 46. This is used to determine the angles of the joints 41a to 43b.

The control device 10 includes an information processing device such as a personal computer or a tablet-type portable information terminal.
As shown in FIG. 1, the control device 10 includes a control unit 11 that controls the entire control device 10, a storage unit 12 that stores programs, data, and the like, a display unit 13 that displays images and the like, And an image generation unit 14 that generates an image to be displayed on the display unit 13 in response to an instruction from the control unit 11. The display unit 13 inputs data such as coordinates and images according to a user's operation on a display screen that displays images and a touch panel or the like. Further, the control device 10 performs a setting for controlling the operation of the robot 1, a voice input unit 15 for inputting a voice, a text input unit 16 for inputting a text, an operation input unit 17 for inputting an operation instruction. And a setting processing unit 18.

The storage unit 12 stores a structure DB (database) 21 in which data indicating the structure of the robot 1 including the above-mentioned DH parameters and the like is stored, and data indicating a correspondence between the structure of the robot 1 and an image of the robot. A correspondence DB 22 and a library DB 23 storing commands and the like for the controller 2 for controlling the operation of the robot 1 are stored. In addition, the storage unit 12 stores an operation DB 24 that stores a correspondence between a robot image and a command in the library DB 23 and a combination DB 25 that stores a combined operation.

The setting processing unit 18 includes an image input unit 31 for inputting an image of the robot from the camera 3 or the display unit 13, an associating unit 32 for associating a region in the image with the configuration of the robot 1, and a robot operation instruction. And an operation setting unit 34 for setting the operation of the robot in accordance with the instruction input. In addition, the setting processing unit 18 includes an operation synthesizing unit 35 that synthesizes the set operation of the robot, and a selecting unit 36 that selects the operation of the robot in accordance with the operation instruction.

<Control processing>
FIG. 4 is a flowchart illustrating an outline of control processing of the robot 1 by the control device 10.
In the control device 10, first, in S11, the image input unit 31 of the processing unit 18 inputs an image of the robot. The image of the robot 1 is an image that can define the correspondence with the operation of the robot 1, such as a handwritten image input by the user on the touch panel of the display unit 13 or an image of the actual robot 1 input via the camera 3. Should be fine. Specifically, for example, a handwritten image simulating the structure of the robot 1 shown in FIG. 5 or an image of the robot 1 captured by the camera 3 can be used.

Next, in S12, the association unit 32 associates the image of the robot with the actual structure of the robot 1 as described above. When a handwritten image is used as an image of a robot as described above, the arrangement of joints and the length of an arm may not exactly match the structure of the robot 1. For this reason, the associating unit 32 associates an area in the image of the robot with the actual structure of the robot 1 stored in the structure DB 21. The association unit 32 stores the result of the association in the association DB 22.

Specifically, as shown in FIG. 6, for example, the associating unit 32 displays an image 51 of the robot on the display screen of the display unit 13 and uses the touch panel or the like of the display unit 13 to define the definition of the association from the user. input. The definition of the association includes, for example, an area selection image 52 for selecting a predetermined area of the target robot image, and a definition information image 53 corresponding to the area selection image 52. FIG. 6 shows an example of an image when the area selection image 52 is associated with the hand unit 44 of the robot 1. When associating with a joint, the user, for example, as shown in FIG. 7, uses an area selection image 54 for selecting a joint and a definition information image 55 corresponding to the area selection image 54. Enter the definition of It should be noted that the definition of the correspondence is not only input as an image, but also as characters corresponding to the voice input by the user via the voice input unit 15, texts input by the user via the text input unit 16, and the like. It may be.

After that, in S13, the operation setting unit 34 sets the operation of the robot 1 corresponding to the image of the robot. The library DB 23 stores, for example, images and keywords corresponding to commands and parameters for operating the robot 1, as shown in FIG. FIG. 8 shows a command (Line) for moving the hand unit 44 of the robot 1 linearly, a command (Move) for moving the hand unit 44 at the highest speed, and respective parameters (movement destination, direction, and movement speed). The example which matched the image and the keyword is shown. The operation setting unit 34 selects a user from a command and parameters indicating the operation of the robot 1 stored in the library DB 23 for a predetermined area in the robot image input by the user via the instruction input unit 33. Is selected, the selected command and parameter are set as an operation corresponding to a predetermined area in the image, and stored in the operation DB 24.

The instruction input unit 33 selects a predetermined area in the image of the robot according to an area input by the user via the touch panel of the display unit 13, for example. In addition, the instruction input unit 33 is, for example, a handwritten character or an image input by the user via the touch panel of the display unit 13, a character corresponding to the voice input by the user via the voice input unit 15, and the text input unit 16. A command and a parameter corresponding to a sentence or the like input by the user are selected from the library DB 23. The recognition accuracy can be improved by learning the recognition of handwritten characters or images, the recognition of voice input by the user, the recognition of keywords in text input by the user, and the like by machine learning.

When inputting a predetermined area, a command, and a parameter by hand, the operation setting unit 34 causes the display unit 13 to display an image 51 of the robot via the image generation unit 14, as shown in FIG. 9, for example. The user who sees this image inputs an area selection image 56 indicating an area and

operation instruction images

57 and 58 through the instruction input unit 33 via the touch panel or the like of the display unit 13. FIG. 9 shows an example in which an instruction to move the position of the hand 44 is input. Further, FIG. 9 uses a square image for selecting the area as the area selection image 56 indicating the area, an arrow image indicating the movement direction as the

operation instruction images

57 and 58 indicating the parameters, and a square image indicating the movement destination. The case where an image is used is shown. When instructing the rotation of the joint, the user inputs a square area selection image 59 indicating a target area and an operation instruction image 60 of an arrow indicating a rotation direction, as shown in FIG. 10, for example. It may be. Alternatively, the area selection image indicating the area may be a character such as “x” or a symbol. When the command and the parameter are input by voice, the instruction input unit 33 causes the voice input unit 15 to recognize the voice input by the user, and stores the command and parameter corresponding to the keyword in the character data of the recognition result in the library DB 23. , And set as an operation corresponding to the image of the robot, and the set result is stored in the operation DB 24.

Further, in S14, the operation setting unit 34 determines whether or not the setting has been completed, and if not completed, repeats the processing of S13. If the setting has been completed, the process proceeds to S15, and the operation using the robot image is started. Specifically, for example, as illustrated in FIG. 11, the selection unit 36 causes the display unit 13 to display the

robot images

61 and 62 stored in the motion DB 24. When the user designates one of the

robot images

61 and 62 using the pointer 65 or the touch panel of the display unit 13, the selection unit 36 reads the operation setting corresponding to the designated robot image from the operation DB 24. Further, the control unit 11 controls the operation of the robot 1 via the controller 2 according to the setting of the operation read by the selection unit 36. In addition to the instruction of the image of the robot, the selection unit 36 may make the voice input unit 15 recognize the user's voice, and give the user's operation instruction according to the recognition result. The selection unit 36 reads, from the operation DB 24, the setting of the operation corresponding to the keyword corresponding to the character data of the recognition result by the voice input unit 15. Specifically, for example, when the character data of the recognition result is “movement”, the setting of the operation corresponding to the robot image 62 is read from the operation DB 24. Alternatively, when the character data of the recognition result is “rotation”, the setting of the operation corresponding to the robot image 61 is read from the operation DB 24. Alternatively, the user may input a handwritten character by operating the touch panel of the display unit 13, and the selection unit 36 may read out the operation setting corresponding to the character data of the recognition result from the operation DB 24. Alternatively, instead of displaying the image of the robot as described above, the display of characters and symbols corresponding to the motion is stored in the motion DB 24 in association with the display, and the display of the characters and symbols is displayed on the display unit 13. Then, the setting of the operation corresponding to the one selected by the user from the display may be read from the operation DB 24.

By the way, in this robot control system, it is possible to combine the settings of a plurality of operations set as the operations corresponding to the images of the robot as described above. Specifically, as shown in FIG. 12, the motion synthesizing unit 35 displays a plurality of

robot images

91, 92,... Stored in the motion DB 24 in association with the motions as described above, on the display unit. 13 in the list display area 82. The motion synthesizer 35 arranges the

images

95, 96,... Selected by the user from the plurality of

robot images

91, 92,. It is displayed in the series display area 81. When the selection by the user is completed, the motion synthesizing unit 35 stores information indicating the images of the selected robots in chronological order in the synthesis DB 25 as a synthesized motion. Thus, in the present embodiment, an operation in which a plurality of operations are combined in time series can be easily set.

When the operation of the robot 1 is controlled in accordance with the synthesized operation set as described above, the selection unit 36 displays the synthesis stored in the synthesis DB 25 on the display unit 13 as shown in, for example, FIG. The

robot images

85, 86,... Corresponding to the performed operations are read out from the operation DB 24 and displayed. When the user designates one of the

robot images

85 and 62 with the pointer 87 or the touch panel of the display unit 13, the selection unit 36 outputs information indicating a combined operation corresponding to the designated robot image to the combination DB 25. Read from Further, the selection unit 36 sequentially reads out the setting of each operation from the operation DB 24 based on the information indicating the images of the plurality of robots in the information in chronological order. The control unit 11 sequentially controls the operation of the robot 1 via the controller 2 according to the setting of the operation sequentially read by the selection unit 36.

<Effects of Embodiment>
As described above, according to the present embodiment, the user can easily set the operation of the robot by setting the operation instruction of the robot corresponding to the image of the robot. In the present embodiment, the user controls the operation by selecting the display indicating the operation instruction corresponding to the instruction input such as the image of the robot and supplying the operation instruction set by the operation setting unit to the robot. be able to. Further, in the present embodiment, by combining a plurality of operation instructions set by the operation setting unit in a time series, it is possible to easily set a synthesized operation.

<Modification>
In the above-described embodiment, an example has been described in which a handwritten image simulating the configuration of a robot is used as an image of the robot. However, any image that can define a correspondence relationship with the operation of the robot 1 is illustrated in FIG. An image 67 of a humanoid robot, an image 71 of the actual robot 1 input through the camera 3 and the like as shown in FIG. Even when such a robot image is used, the operation of the robot can be easily set by performing the same operation setting as described above.

FIG. 14 shows a case where a rectangular image is used as the area selection image 69a for selecting the hand unit 44 of the robot 1, and characters indicating arrows and parameters are used as the

operation instruction images

69b and 69c. FIG. 15 uses a square image as the region selection image 72 for selecting the hand unit 44 of the robot 1, and uses arrow images indicating the direction of movement and square images indicating the destination as the

operation instruction images

73 and 74. The case is shown.
The above embodiment is an example as a means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the apparatus or system to which the present invention is applied or various conditions. However, the present invention is not limited to this.

DESCRIPTION OF SYMBOLS 1 ... Robot, 2 ... Controller, 3 ... Camera, 11 ... Control part, 12 ... Storage part, 13 ... Display part, 14 ... Image generation part, 15 ... Voice input part, 16 ... Text input part, 17 ... Operation input part , 18 setting processing unit, 21 structure DB, 22 correspondence DB, 23 library DB, 24 operation DB, 25 synthesis DB, 31 image input unit, 32 association unit, 33 instruction input unit, 34: motion setting unit, 35: motion synthesis unit, 36: selection unit, 41a, 41b, 42a, 42b, 43a, 43b: joint unit, 44: hand unit, 45, 46, 47: arm

Claims

A control device for controlling a robot,
A structure database that holds the structure of the robot to be controlled,
An image input unit for inputting an image of the robot,
An association unit that associates a predetermined area in the image of the robot input by the image input unit with the structure of the robot held in the structure database;
A library database that stores an operation instruction to the robot and an instruction input corresponding to the operation instruction;
An instruction input unit for inputting a predetermined area in the image of the robot and an instruction input corresponding to the area,
An operation setting unit that sets the operation instruction in the library database corresponding to the instruction input input by the instruction input unit as the operation instruction corresponding to the instruction input;
A control device comprising:
A selection unit for displaying a display indicating an operation instruction corresponding to the instruction input set by the operation setting unit, and when the display is selected, supplying the operation instruction to the robot;
The control device according to claim 1, further comprising:
The library database holds a keyword corresponding to the operation instruction,
The instruction input unit outputs a character corresponding to the instruction input,
The operation setting unit performs the setting according to a character from the instruction input unit and a keyword in the library database.
The control device according to claim 1 or 2, wherein:
The library database holds images corresponding to the operation instructions,
The instruction input unit outputs an image corresponding to the instruction input,
The operation setting unit performs the setting according to the image from the instruction input unit and the image in the library database.
The control device according to claim 1 or 2, wherein:
The instruction input unit,
Outputting an image indicating a component of the robot and an image indicating a destination of the component as an image corresponding to the instruction input;
The control device according to claim 4, wherein:
A combining unit that combines the plurality of operation instructions set by the operation setting unit in a time-series manner,
The control device according to any one of claims 1 to 5, further comprising: