JP2017113815A - Image display method of robot system that holds object using robot - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information display device that is configured to display an image suitable for visually recognizing a work environment so as to allow a visual burden of a user to be reduced, regardless of a direction in which an imaging device is installed and a position where a display monitor is installed.SOLUTION: The robot system, which is configured to hold an object using a robot, comprises: an imaging camera or a measuring sensor for recognizing the object; display means that displays on a monitor information imaged by the camera or information measured by the sensor; converting means that converts uniformly an orientation of information to be displayed by the display means; and designating means that designates a parameter for the converting means. The converting means executes a rotational conversion or a reverse conversion or both conversions according to the parameter.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image display device for a robot system that measures the position / posture of an object using an imaging device, grips a component using a robot based on the information, and supplies, conveys, assembles, and assembles the component, And programs.

  In recent years, in a manufacturing site such as a factory, a robot system including a robot arm or the like is often assembled and transported instead of a human. For example, a captured image of a manufactured part or a tray for storing the part is acquired using an imaging camera, and the robot arm is moved to a position suitable for gripping the manufactured part by analyzing the captured image. I can do it.

  Further, as disclosed in Patent Document 1, a plurality of manufacturing parts, robot arms, and the like are provided so that the recognition accuracy of the position and orientation (orientation) of the manufacturing parts can be improved and the robot can accurately hold the manufacturing parts. There is a robot system including a setup process in which a posture is imaged in advance and an analysis result is fed back as gripping posture information.

JP2012-101320A

  In a scene where a robot system that controls a robot based on such a captured image is constructed, or in a scene where the system is confirmed to be operating correctly, the operator refers to the imaging result on the display monitor and simultaneously It is common to perform this while visually checking the current working environment of the robot system.

  Further, after the system construction is completed, it is effective to visually recognize the operation status of the robot system to a third party by comparing various images displayed on the display monitor with the work environment.

  However, the mounting direction of the imaging device, the position where the display monitor is installed, and the orientation of the work environment directly viewed by the user including the worker and the observer vary depending on the viewing direction, and the orientation and visual recognition of the image generated by the imaging device. In many cases, the orientations of the work environments do not match, which complicates the visual recognition of the robot system and increases the burden on the user.

In order to solve the above problems, a robot system according to the present invention includes:
In a robot system that grips a target object using a robot,
The system includes a photographing camera or a measurement sensor for recognizing the target object,
Display means for displaying information taken by the camera or information measured by the sensor on a monitor;
Conversion means for uniformly converting the orientation of information displayed on the display means;
Specifying means for specifying parameters of the conversion means,
The conversion means is a conversion that performs rotation conversion and / or inversion conversion according to the parameter.

  According to the robot system of the present invention, it is possible to reduce the burden on the user when visually recognizing the complicated work procedure adjustment of the robot system and to improve the visibility of the operation confirmation of the operating robot system system. It is done.

It is a figure which shows the structure of the robot system in 1st embodiment. It is a figure which shows the information processing of the robot system applied to 1st embodiment. It is an example of the setting screen of the image display azimuth | direction of the robot system concerning this invention. It is the figure which illustrated the setting method of the image display direction concerning this invention. It is a flowchart which shows the flow of the production | generation / update process of a target object image. It is a figure which shows the structure of the robot system applied to 2nd embodiment. It is a figure which shows the structure of the robot system applied to 3rd embodiment. It is a figure explaining the azimuth | direction adjustment method when the apparent azimuth | direction of a work environment is reversed.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<First Embodiment>
An apparatus configuration of a suitable robot system in the first embodiment will be described with reference to various drawings. In addition, the same symbol is appended to the same member / process described in the drawings, and the re-explanation of the symbol already described may be omitted.

  FIG. 1 is a diagram showing a configuration of a robot system (object gripping system) in the present embodiment. 1A is a view of the configuration of the robot system as viewed from above, and FIG. 1B is a view of the configuration of FIG. 1A as viewed from the side.

  Reference numeral 100 denotes a work environment 100 indicating a state and a space area for the robot system to assemble and transport manufactured parts. The robot system includes a robot arm 11, an imaging device 13, and a controller 14.

  The robot arm 11 is an arm having six degrees of freedom and includes a two-finger hand mechanism (hereinafter referred to as “hand 12”) as an end effector. The hand 12 has a structure that can grip the component 16. The combination of the robot and the hand should be designed in accordance with the shape characteristics of the object to be grasped, and is not limited to the 6-axis robot and the two-finger hand mechanism shown here.

  Moreover, although the component 16 illustrated the eyebolt in FIG. 1, other components, such as a screw, may be sufficient. The components 16 are initially loaded in a disjointed posture on the component tray 15, and the components 16 determined to be grippable by measuring the position and orientation are gripped by the robot arm 11. Furthermore, the parts 16 are arranged in the position / orientation (orientation) designated by the controller 14 on the work table 17 for assembling, transporting, and processing, or assembled in other mechanisms. An example of the arrangement of the components 16 arranged on the workbench 17 is indicated by a broken line in FIG. In addition, the components 16 may be arranged and placed in the component tray 15 and may be held using a robot and a hand.

  The imaging device 13 is a camera. The imaging device 13 acquires image data for measuring the position of the component by imaging the component 16 or imaging the component tray 15 which is a stored item. This imaging apparatus may be a collective information acquisition method using a two-dimensional sensor array or may be a scan method. The information obtained by the measurement is two-dimensional luminance distribution information that extends in the direction perpendicular to the optical axis, or the measurement results are collected for each illumination that is patterned by applying the triangulation method. Information on the coordinates of the generated three-dimensional object may be used. Further, it may be coordinate information of a three-dimensional object obtained by referring to the defocus characteristic of the optical system.

  The frame 18 is a structure for fixedly mounting the imaging device 13. The frame 18 may be a ceiling, a floor, a wall or the like depending on an object to be imaged.

  The controller 14 controls the operation of each mechanism by controlling the robot arm 11, the hand 12, and the camera 13. Here, the controller 14 is generally configured by a computer, but the present embodiment does not limit the configuration. A component tray 15 is loaded with components 16 for assembly and conveyance.

  An image as an imaging result obtained during the configuration or operation of the robot system, or distribution data as an analysis result of the image (hereinafter simply referred to as “image”) is displayed on the display monitor 19. The display monitor 19 is directed in the direction of the arrow added in FIG. The operator 20 confirms the display monitor 19 and the work area 100 in the direction shown in the figure, and performs setting and operation of the robot system.

  FIG. 2 is a block diagram illustrating an example of a schematic configuration of the controller 14.

  The bus 200 includes a CPU 201, a ROM 202, a RAM 203, an image processing circuit 204, a nonvolatile large-capacity memory 205, an imaging control unit (I / F) 206, an image input unit (I / F) 207, and a robot control unit (I / F) 208 and image output unit (I / F) 209 are connected. The ROM 202 stores a boot program for the CPU 201, constant parameters, and the like. The RAM 203 is used as a work area for the CPU 201 and a storage area for temporarily storing images picked up by the image pickup device 13.

  The image processing circuit 204 performs image generation processing to be referred to when the robot arm 11 grips the component 16, posture pattern detection processing for estimating the position and posture of the object from an image obtained by capturing the component tray 15 on which the component is mounted, And a dedicated hardware circuit for processing at least a part of the preprocessing necessary for them. Note that the image processing circuit unit 204 is not particularly provided in the system, and all image processing can be performed by a program operated by the CPU 201. In this case, normally, the cost can be reduced, but the processing speed becomes slow.

  The nonvolatile large-capacity memory 205 is, for example, an HDD device, and stores a program operated by the CPU 201, system parameters, and an image of the generated object.

  The imaging control unit 206 includes a dedicated I / F that is connected to the imaging device 13 and performs imaging control such as notification of imaging timing.

  The image input unit 207 is connected to an output unit (not shown) of the controller 14 and takes captured image data into the controller.

  The robot control unit 208 is connected to the robot arm 11 and the hand 12 and includes an I / F for performing desired operation control.

  The image output unit 209 is connected to an input unit (not shown) of the display monitor 19 and displays an image and parameters obtained by imaging and analysis on the display monitor 19.

  Next, a method for matching the image on the display monitor and the appearance (orientation) of the worker 20 in the work environment 100 in this embodiment will be described with reference to FIGS.

  FIG. 3 is a UI (user interface) display orientation setting screen 300 that can be referred to on the display monitor 19 by the operator 20, and the orientation of the image 301 can be rotated by a specified angle.

  The display orientation setting screen 300 can transition from the operation environment setting screen of the controller 14. The display orientation setting screen 300 shows an image 301 acquired by the imaging device 13. The initial value of the orientation of the image 302 remains the orientation of the image held inside the controller 14. The rotation of the image 301 can be changed at any time by the direction defining means 302.

  The azimuth defining means 302 includes a left 90 degree rotation button 302-A, a right 90 degree rotation button 302-B, and an azimuth input box 302-C. When the first two buttons are pressed, the image 301 immediately updates the display by rotating the image as indicated by each button. When the image 301 is dragged with the mouse, the image 301 is rotated in the display screen in conjunction with the operation, and the azimuth at that time is reflected in the azimuth input box 302-C. It is also possible to directly input a numerical value (angle) in the azimuth input box 302-C and rotate the image. The numerical value (rotation angle) directly input is reflected as the screen rotation angle of the two-dimensional image by pressing the “apply” button 303.

  In the image 301, the axes (+ x axis, + y axis) for indicating the orientation are shown superimposed, and the orientation of these axes is also updated when the screen is rotated in accordance with the change of the orientation.

  In order to reflect the rotation of the screen changed on the display orientation setting screen 300, an “apply” button 304, an “OK” button 305, a “default” button 306, and a “cancel” button 307 are used.

  When the “apply” button 304 is pressed, the orientation display setting of the robot system is saved, and when the “OK” button is pressed, the display orientation setting screen 300 is exited in addition to the function of the “apply” 305 button. When the “default” button 306 is pressed, the change in the azimuth display setting is discarded and the initial state is restored. When the “Cancel” button 307 is pressed, the display direction setting screen 300 is exited without changing the direction display setting.

  When the display orientation is updated, the controller 14 is programmed to display an image held by the robot system on the display monitor 19 and an image rotated by a specified value from the orientation held by the controller 14 each time.

  The method for defining and reflecting the direction using the UI has been described above. Of course, these methods are not limited to those described here, and can take various forms.

  Note that when the image 301 is, for example, data indicating the three-dimensional coordinates of the surface of the imaging target and two-dimensional display is performed by projection, the orientation of the screen rotation remains as it is in the entire area of the default two-dimensional display screen 301. It will be rotated. Since the data indicating the above three-dimensional coordinates can be rotated three-dimensionally by dragging the mouse or the like during display, for example, the three-dimensional coordinate data can be observed even if the setting of the image orientation is changed. Although the viewpoint variation does not increase, there is an effect of reducing the amount of operation until the operator 20 reaches the desired observation viewpoint.

  FIG. 4 illustrates the viewpoint of the display monitor 19 and the work environment 100 that the operator 20 refers to using the UI shown in FIG. When the component tray 15 exists in the imaging range of the imaging device 13, the arrangement on the display monitor 19 is compared with the arrangement of the component tray 15 and the component 16 arranged in the work environment 100.

  FIG. 4A shows a state in which the worker 20 directly visually recognizes the component tray 15 and the component 16 arranged there as a part of the work environment 100. FIG. 4B is an image 301 shown on the display orientation setting screen 300 corresponding to the observation target in FIG. The operator 20 compares these two arrangements and corrects the orientation of the image 302 so that the orientations of rotation match. The correction angle is such that, for example, the part 16 surrounded by the broken line of the circle comes to the lower part of the screen, so the image 301 may be rotated 90 degrees to the left on the paper. FIG. 4C shows the state of the image 301 after rotating the image and setting the azimuth.

  Finally, the processing flow controlled by the controller 14 to correctly reflect the rotation instruction amount of the image display orientation set by the operator 20 will be described using each step number shown in FIG. 5 (S indicates a step).

  In step S501, the setting value of the azimuth is acquired in advance when the command for displaying the image 203 is received.

  The subsequent steps are executed whenever there is a display request for the image 203 of the controller 14.

  In step S502, the information of the target image 203 to be displayed is held without considering the rotation.

  In step S503, the image acquired in step S502 is rotated based on the orientation obtained in step S501 or step S505 described later, and displayed on the display monitor 19.

  Step S504 is a step of entrusting a determination as to whether the image 203 on which the worker 20 is displayed matches the orientation when the worker 20 looks at the work environment. If the operator 20 instructs the correction of the direction with the UI, a setting screen as illustrated in FIG. 3A is opened, and the direction is corrected in step S505.

  In step S506, the operator 20 instructs to change the type of the image 203. In step S507, the request is reflected. At this time, the original image 203 and the rotation direction remain the same.

  This is the processing flow related to the display of the image 203 by the controller 14. This flow continues until the operator 20 proceeds to step S508 instructing to end the display of the image 203.

  When the controller 14 accepts the instruction in step S508 and proceeds to step S509, the controller 14 stores the current orientation setting. Finally, the display of the image 203 is terminated in step S510.

Second Embodiment
An apparatus configuration of a suitable robot system in the second embodiment will be described with reference to various drawings.

  The difference between the present embodiment and the first embodiment is that there are a plurality of humans who visually recognize the robot system, and the azimuths for visually recognizing the work environment are different.

  FIG. 6 is a view of the apparatus configuration of the robot system according to the second embodiment as viewed from above. The robot system according to the first embodiment has the following changes. That is, the observer 22 who visually recognizes the work environment 100 in the direction of the line of sight B from a viewpoint different from the line of sight A when the worker 20 visually recognizes the work environment 100, and the observer 22 as a second image display device The difference is that there is a display monitor 21 for referring to an image acquired by the imaging device 13. The system of the operation screen displayed on the display monitor 21 is the same as the system displayed on the display monitor 19. Further, the layout of the frame and the controller 14 is different so that the observer 21 can refer to the imaging range of the imaging device 13, but the functions performed by the imaging device 13 and the frame 18 are the same. The image output unit 209 provided in the controller 14 is configured to output images to two display monitors, that is, the display monitor 19 and the display monitor 21.

  The operator 20 can easily view the construction of the robot system or the operating state of the system, and the orientation of the image 301 on the display monitor 19 corresponding to the line of sight A and the image 301 on the display monitor 21 corresponding to the line of sight B. Set and save each direction. At this time, since the directions of the line of sight A and the line of sight B are different, the set values of the orientations of the images 301 corresponding to the respective lines of sight are different.

  In this way, if the two display monitors separately specify the directions in which various images are rotated, for example, the operator 20 observes the robot system from the direction of the line of sight A, and another person from the direction of the other line of sight B, That is, the observer 22 can also easily see the robot system.

  As described above, in this embodiment, the method for different humans to visually recognize the robot system from different viewpoints has been described. However, in the present embodiment, a method that makes it easy to visually recognize the work environment from a plurality of viewpoints using a plurality of display monitors. Explains. Therefore, it is not necessary to actually distinguish at least two persons, that is, persons corresponding to the worker 20 and the observer 22.

  For example, the observer 22 may be a person who adjusts the setting of the robot system, like the operator 20, and the operator 20 visually recognizes the robot system as the real observer 22 when the robot system is ready for operation. It will be.

  Furthermore, the worker 20 and the observer 22 may be one person, and one person moves to the position of the worker 20 or the position of the observer 22, and constructs and operates the robot system while changing the viewpoint. It is also possible to carry out with reference to the display monitor at the position.

  Of course, three or more display monitors may be prepared, and each display monitor may be arranged at a position where the work environment 100 can be viewed from different directions.

<Third Embodiment>
A configuration of a preferred robot system in the third embodiment will be described with reference to various drawings.

  FIG. 7 is a diagram showing a device configuration of the robot system in the present embodiment.

  This configuration is the same as that shown in FIG. 1 except that an opaque plate-like protective fence 23 and a mirror 24 are added to protect the user from a robot arm or the like.

  If the robot's protective fence 23 is necessary and it is difficult for the operator 20 to visually recognize the work environment 100 directly, it is necessary to visually inspect the robot system indirectly using the mirror 24. The mirror surface of the mirror 24 may be a flat surface or a curved surface, and may be equipped with a scanning mechanism so as to cover a range to be visually recognized by the robot system.

  When indirectly viewing the work environment 100 as shown in FIG. 3, it is often the case that an image that is inverted vertically, horizontally, or diagonally with respect to the appearance of direct viewing is referred to.

  FIG. 8 shows how the image 301 and the worker 20 in the work environment 100 are seen when the image of the work environment 100 visually recognized as shown in FIG. It is a figure for demonstrating the method to match (azimuth | direction).

  FIG. 8A shows a situation when the worker 20 indirectly visually recognizes the component tray 15 and the components 16 arranged there as a part of the work environment 100. In this orientation, the coordinates of the apparent arrangement of the component 16 are inverted up and down by the mirror 24 as compared with the arrangement when the mirror 24 is not shown in FIG. That is, the apparent orientation of the work environment 100 is inverted upside down.

  FIG. 8B shows a result of adjusting the orientation of the work environment 100 obtained by the above visual recognition using the display orientation setting screen 300. In FIG. 8B, the direction of the y-axis is the same, and only the positive / negative of the x-axis is inverted when compared with FIG. 4C. In other words, the orientation of the image 301 can be adjusted by a combination of coordinate rotation of the screen and coordinate inversion in a specific direction (for example, the x axis).

  Accordingly, as the UI of the display orientation setting screen 300 of this embodiment, for example, a button in which a button for inverting the coordinates only in the x-axis direction is added together with the UI configuration exemplified in the first example is used.

  The direction for reversing the coordinates does not have to be the x-axis, and may be a direction having an angle with respect to the x-axis. Even if UIs having different inversion directions are used, there is a rotation angle that always results in the same direction conversion. In addition, the order of the coordinate inversion and rotation may be first (however, the rotation angle depends on the order).

  As described above, the operator 20 can match the orientation of the display monitor 19 and the apparent work environment 100 by setting the rotation or inversion of the orientation of the image 301, Visual recognition becomes easy.

  The present invention is a robot system that measures the position / posture of an object with an imaging device, grips a part using a robot based on the information, and supplies, conveys, assembles, and assembles the part, particularly a manufactured part. The present invention can be applied to a component picking system for the purpose of assembly and transportation.

100 working environment, 11 robot arm, 12 hand, 13 imaging device,
14 controller, 15 parts tray, 16 parts, 17 workbench,
18 frames, 19 display monitors, 20 workers, 21 display monitors, 22 observers,
200 bus, 201 CPU, 202 ROM, 203 RAM,
204 image processing circuit, 205 non-volatile large-capacity memory,
206 Imaging control unit (I / F), 207 Image input unit (I / F),
208 Robot control unit (I / F), 209 Image output unit (I / F),
300 display orientation setting screen, 301 image, 302 orientation defining means,
302-A Left 90 degree rotation button, 302-B Right 90 degree rotation button,
302-C Direction input box, 303 Apply button,
304 Setting update button

Claims (1)

In a robot system that grips and picks a target object using a robot,
The system includes a photographing camera or a measurement sensor for recognizing the target object,
Display means for displaying information taken by the camera or information measured by the sensor on a monitor;
Conversion means for uniformly converting the orientation of information displayed on the display means;
Specifying means for specifying parameters of the conversion means,
The robot system according to claim 1, wherein the conversion means is a conversion that performs rotation conversion and / or reverse conversion according to the parameter.