JP2021181134A - Operation assisting device for work robot, method and program - Google Patents

Abstract

To provide an operation assisting technique for a work robot, by which the shape of a structure in a work area can be recognized with high reliability.SOLUTION: An operation assisting device 10 includes: a reception unit 11 that sequentially receives measurement signals 26 of a structure 25, measured by a shape measurement sensor 21 in a work area 20; a coordinate conversion unit 12 by which a point group data 27 including a three-dimensional shape of the structure 25 reproduced from the measurement signals 26 is converted into a coordinate system of the work area 20 from a coordinate system of the shape measurement sensor 21; a development unit 15 by which three-dimensional data 28 indicating a three-dimensional shape of a work robot 22 is developed into the coordinate system of the work area 20, based on control information 31; a detection unit 16 that detects an overlap area 35 of the point-group data 27 and the three-dimensional data 28 in the coordinate system of the work area 20; and a generation unit 17 that updates the point-group data 27 in an update area excluding the overlap area 35 and generates a shape map 37 for the work area 20.SELECTED DRAWING: Figure 1

Description

An embodiment of the present invention relates to an operation support technique for a remotely controlled work robot.

Remote control of a work robot such as an articulated manipulator is performed while checking the work state with a surveillance camera in order to avoid collision with the target structure and its surrounding structures. Therefore, depending on the posture of the working robot during operation, the field of view of the surveillance camera may be obstructed and a blind spot may be generated, which may make it difficult to confirm the working state.

The following publicly known techniques have been proposed as a method for eliminating the blind spot in the field of view of such a surveillance camera. First, a publicly known technique has been proposed in which a plurality of surveillance cameras are installed in a work area and a sub-image of another surveillance camera is synthesized in a blind spot area of the main image of the surveillance camera. Secondly, there has been proposed a publicly known technique for planning an operation in a virtual space in which a work robot does not collide based on a three-dimensional shape of a structure arranged in a work area. As this three-dimensional shape, actual measurement data and design data in the work area are used.

Japanese Unexamined Patent Publication No. 2006-53922

However, in the first known technique described above, the number and positions of surveillance cameras that can be installed in the work area may be limited due to restrictions on the work environment. In addition, synthesizing a sub-image in the blind spot area of the main image causes images with different viewpoints to be fitted, so that the continuity of the posture of the reflected structure is lost, and there is a concern that erroneous operation may occur in remote work. be. This concern becomes more pronounced as the distance between multiple surveillance cameras is increased so that the blind spots of each other can be efficiently complemented.

In the second known technique described above, when design data is used as a three-dimensional shape, the shape of the structure arranged in the work area may differ from the actual shape, or the structure may be transported or removed. There is a problem of lack of reliability, such as being unable to respond to changes. One of the solutions to this problem is the use of measured data in the work area. However, when a part of the work robot is included in the measurement area of the measured data, there is a problem that the shape of the structure arranged in the work area cannot be recognized.

An embodiment of the present invention has been made in consideration of such circumstances, and an object of the present invention is to provide an operation support technique for a work robot that can recognize the shape of a structure in a work area with high reliability.

In the operation support device of the work robot according to the embodiment, the receiving unit that sequentially receives the measurement signal of the structure measured by the shape measurement sensor in the work area and the three-dimensional shape of the structure reproduced from the measurement signal. The point cloud data including the point cloud data is converted from the coordinate system of the shape measurement sensor to the coordinate system of the work area, and the three-dimensional data showing the three-dimensional shape of the work robot is the coordinates of the work area based on the control information. The expansion unit developed in the system, the detection unit that detects the overlapping area between the point cloud data and the solid data in the coordinate system of the work area, and the point cloud data are updated in the update area excluding the overlapping area. It includes a generation unit that generates a shape map of the work area.

An embodiment of the present invention provides a work robot operation support technique capable of recognizing the shape of a structure in a work area with high reliability.

The block diagram of the operation support apparatus of the work robot which concerns on 1st Embodiment of this invention. A block diagram showing an initialized work robot. A three-dimensional shape represented by point cloud data measured by a camera immediately after initializing the work robot. A three-dimensional shape represented by point cloud data measured by a camera during the operation process of a work robot. A shape map of the work area generated during the operation process of the work robot. The block diagram of the operation support apparatus of the work robot which concerns on 2nd Embodiment of this invention. The flowchart explaining the operation support method of the work robot and the embodiment of the operation support program.

(First Embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a configuration diagram of an operation support device 10A for a work robot according to the first embodiment of the present invention. In this way, the operation support device 10A (10) is a measurement signal that is a receiving unit that sequentially receives the measurement signal 26 of the structure 25 (25a, 25b) measured by the camera 21 that is the shape measurement sensor in the work area 20. The receiving unit 11, the coordinate conversion unit 12 in which the point group data 27 including the three-dimensional shape of the structure 25 reproduced from the measurement signal 26 is converted from the coordinate system of the camera 21 to the coordinate system of the work area 20 and the work. The expansion unit 15 in which the three-dimensional data 28 showing the three-dimensional shape of the robot 22 is expanded in the coordinate system of the work area 20 based on the control information 31, and the point group data 27 and the three-dimensional data 28 in the coordinate system of the work area 20. A generation unit that updates the point group data 27 in the overlap area detection unit 16 that detects the overlap area 35 and the update area 36 (FIG. 5) excluding the overlap area 35 and generates the shape map 37 of the work area 20. The shape map generation unit 17 is provided.

In the present embodiment, an articulated manipulator is exemplified as the work robot 22, and the work of grasping and moving the target structure 25a (structure 25) to be worked in the work area 20 is exemplified. The work robot 22 is remotely controlled by an operator (not shown), and the operator grasps the work status of the target structure 25a via a moving image from the camera 21.

The work robot 22 is provided with a gripper 34 at the hand and is arranged in the work area 20. Although one end of the work robot 22 is fixed to the floor surface of the work area 20, the installation form thereof is not limited. Further, an articulated manipulator can be installed on a trolley (not shown) and moved to an arbitrary position in the work area 20. In this case, it is necessary to measure the moving position of the dolly in the coordinate system of the work area 20 and reflect the position information of the work robot 22 such as a manipulator in the processing of the developing unit 15.

In the work area 20, in addition to the target structure 25a, a peripheral structure 25b is also arranged as the structure 25. The operator (not shown) operates on the target structure 25a while paying attention to the mechanical interference between the peripheral structures 25b and the working robot 22.

The camera 21 is installed in the work area 20 and is a device capable of measuring the three-dimensional shape of the structure 25. Examples of the camera 21 include a camera that measures a shape by combining a plurality of optical imaging devices and a stereoscopic view, and a camera that measures a shape by a laser range finder or a distance image sensor. Shape measurement is a three-dimensional shape measuring device. A general sensor can be adopted.

Then, the camera 21 processes the measured surface shape of the structure 25 as point cloud data in the camera coordinate system, puts it on the measurement signal 26, and sequentially transmits it. Then, the measurement signal receiving unit 11 sequentially receives the measurement signal 26 transmitted from the camera 21, and when the working robot 22 or the structure 25 is moving, the point cloud data 27 is captured as a moving image.

The coordinate conversion unit 12 converts the point cloud data 27 defined in the coordinate system of the camera 21 in the measurement signal 26 into the coordinate system of the work area 20. Here, the coordinate system of the work area 20 reproduces the point cloud data 27 of the structure 25 received from the camera 21 and the three-dimensional data 28 of the work robot 22 developed from the control information 31 in a common three-dimensional space. It is a coordinate system for. As will be described later, not only the structure 25 but also a part of the working robot 22 may be reflected in the point cloud data 27 as a three-dimensional shape in the working area 20.

The working robot 22 is composed of a plurality of links 38, a servomotor 39 that connects the ends of the respective links 38 and functions as joints, and a gripper 34 for gripping the target structure 25a. The work robot control unit 30 controls the operation of the servomotors 39 constituting each joint according to a predetermined motion plan or an operation command of an operator (not shown), and controls the posture of the work robot 22.

The work robot control unit 30 directly sends a control signal 33 for driving the servomotor 39 and the like to the work robot 22, and also sends control information 31 indicating the drive amount of the servomotor 39 to the operation support device 10. Is also sent. The control information 31 may be directly transmitted from the work robot 22 instead of the work robot control unit 30.

The unfolding unit 15 is a tertiary of the work robot 22 from the robot part shape model 32 which is a shape model of the parts of the work robot 22 (for example, the link 38), the control information 31 of the parts of the work robot 22 (for example, the servo motor 39), and so on. The three-dimensional data 28 showing the original shape (attitude) is virtually reproduced. The robot part shape model 32 can be prepared in advance using design information such as three-dimensional CAD data, and the control information 31 can use real-time angular displacement data in the servomotor 39. As a result, the three-dimensional data 28 is expanded while being sequentially updated in the coordinate system of the work area 20 shared with the point cloud data 27, and the movement is virtually reproduced.

FIG. 2 is a configuration diagram showing an initialized work robot 22. FIG. 3 is a three-dimensional shape represented by the point cloud data 27 measured by the camera 21 immediately after the working robot 22 is initialized. Initialization means that the working robot 22 takes a posture of retracting from the measurement area of the camera 21.

FIG. 4 is a three-dimensional shape represented by the point cloud data 27 measured by the camera 21 in the operation process of the work robot 22. As described above, when the working robot 22 enters between the camera 21 and the structure 25, the acquired point cloud data 27 includes the three-dimensional shape of the working robot 22. That is, in the measurement area of the camera 21, the three-dimensional shape of the structure 25 that is the blind spot of the work robot 22 is not reflected in the point cloud data 27.

From the point cloud data 27 itself reproduced from the measurement signal 26, it is not possible to distinguish between the three-dimensional shape corresponding to the structure 25 and the three-dimensional shape corresponding to the working robot 22. Therefore, when taking an image with the camera 21, the work robot 22 is initialized before the start of the image pickup, and the point cloud data 27 that accurately reflects the arrangement of the structure 25 in the work area 20 is acquired.

The explanation will be continued by returning to FIG.
When the work robot 22 is reflected in the measurement area of the camera 21 in the operation process of the work robot 22, an overlapping area 35 of the point cloud data 27 and the three-dimensional data 28 is generated in the coordinate system of the work area 20 in this reflected portion. do. The overlapping area detection unit 16 collates the point cloud data 27 and the three-dimensional data 28 in the coordinate system of the work area 20, and detects the overlapping area 35 of both.

FIG. 5 is a shape map 37 of the work area 20 generated in the operation process of the work robot 22. Here, in the measurement signal 26 that is sequentially received, the frame is transferred from the point cloud data 27 of only the structure 25 as shown in FIG. 3 to the point cloud data 27 in which the working robot 22 is reflected as shown in FIG. Is switched. The overlapping region 35 is detected for the first time at the timing of switching from FIG. 3 to FIG.

The explanation will be continued by returning to FIG.
The shape map generation unit 17 uses the point cloud data 27 that constitutes the latest timing frame as shown in FIG. 4 as the update area 36 (FIG. 5) excluding the overlapping area 35. Then, only the part of the update area 36 is overwritten and updated with respect to the point cloud data 27 of the frame at the timing shown in FIG. 3 immediately before that.

As a result, as shown in FIG. 5, the update area 36 is composed of the latest timing point cloud data 27, and the overlapping area 35 is generated as a shape map 37 composed of the point cloud data 27 of the immediately preceding timing. .. As a result, even when the work robot 22 is reflected in the measurement area of the camera 21, the three-dimensional shape of the structure 25 in the blind spot can be grasped.

Further, it is assumed that the work robot 22 grips and moves the target structure 25a, which involves the movement of the three-dimensional shape of the structure 25. Even in this case, the point cloud data 27 in the update area 36, which does not enter the blind spot of the work robot 22, is updated to the latest state. Therefore, the target structure 25a that is gripped and moved can be accurately tracked by the image, and the operation of the working robot 22 can be appropriately performed.

The monitor 18 displays a moving image of the shape map 37. The operator operates the work robot 22 while visually observing the moving image of the monitor 18. Further, the shape map 37 is fed back to the work robot control unit 30, and in the vicinity of the area where the work robot 22 becomes a blind spot and cannot be measured, the operation control of the work robot 22 such as operating the work robot 22 at a low speed is enhanced. Involved in.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 6 is a configuration diagram of the operation support device 10B of the work robot according to the second embodiment of the present invention. In FIG. 6, the parts having the same configuration or function as those in FIG. 1 are indicated by the same reference numerals, and duplicate description will be omitted.

The operation support device 10B (10) of the second embodiment is added to each point group data 27 constituting the shape map 37 in addition to the configuration of the operation support device 10A (FIG. 1) of the work robot of the first embodiment. The elapsed time calculation unit 41, which is a calculation unit that calculates the elapsed time 46 based on the associated camera measurement time 45, and the reliability evaluation unit 42 that evaluates the reliability of each point group data 27 based on the elapsed time 46. And have.

The shape map generation unit 17 associates each of the point cloud data 27 constituting the shape map 37 with a camera measurement time 45 when measured by the camera 21. Then, the elapsed time calculation unit 41 acquires the camera measurement time 45 associated with each point cloud data 27 constituting the shape map 37, takes the difference from the current time, and calculates the elapsed time 46.

The elapsed time 46 is substantially zero for the point cloud data 27 belonging to the update region 36, and the point cloud data 27 belonging to the overlapping region 35 is longer as it continuously belongs to the overlapping region 35 for a long period of time. Become. That is, it can be said that the region of the shape map 37 composed of the point cloud data 27 having a long elapsed time 46 reflects the old working state and has low reliability.

The reliability evaluation unit 42 evaluates the reliability of each point cloud data 27 based on the elapsed time 46. The reciprocal of the elapsed time 46 is defined as the reliability evaluation parameter 47. Based on the evaluation parameter 47 defined in this way, it is clearly shown in the area of the shape map 37 displayed on the monitor 18 that the point cloud data 27 has not been updated for a long period of time, and the operator is alerted. Can be done.

Further, the evaluation parameter 47 is fed back to the work robot control unit 30, and in the vicinity of the area where the work robot 22 becomes a blind spot and cannot be measured, the work robot 22 is operated at a low speed to perform control with enhanced safety. Further, when the evaluation parameter 47 falls below the reference value and the reliability drops to a level that cannot be overlooked, the working robot 22 is initialized and the working robot 22 is retracted from the measurement area of the camera 21. As a result, the overlapping region 35 is eliminated in the shape map 37, the entire region becomes the updated region 36, the evaluation parameters 47 in all the point cloud data 27 are improved, and the reliability is restored.

Further, in the operation support device 10B of the second embodiment, the camera 21 is supported by an adjustment mechanism unit 48 that can freely adjust the measurement area. An articulated arm mechanism is exemplified as the adjusting mechanism portion 48, but the mechanism is not particularly limited, and a mechanism having pan / tilt adjustment and a linear motion moving mechanism may be adopted.

The adjustment mechanism unit 48 inputs a drive signal 49 output from the camera position drive unit 40, and adjusts the position and angle of the camera 21 in the work area 20. The camera position driving unit 40 receives the evaluation parameter 47 of the point cloud data 27 from the reliability evaluation unit 42 so that the surface of the structure 25 corresponding to the unreliable point cloud data 27 is included in the measurement area. Adjust the position and angle of the camera 21. In this way, the evaluation parameter 47 changes the measurement area of the camera 21 in order to improve the reliability of the point cloud data 27.

Further, the operation support device 10B of the second embodiment can evaluate the reliability of each point cloud data 27 based on the distance between the point cloud data 27 of the structure 25 and the three-dimensional data 28 of the work robot 22. can. The work robot 22 has a wide operating range in the work area 20, but the work area actually used is often limited. In particular, when the work robot 22 works in close proximity to the target structure 25a, the work area is limited to the vicinity of the target structure 25a.

Therefore, the reliability evaluation parameter 47 is set lower as the point cloud data 27 closer to the work position of the work robot 22 is more important, and conversely, the evaluation parameter of the point cloud data 27 farther from the work position is set higher. ..

The measurement area by the camera 21 whose position and angle are fixed is limited in the large work area 20. Therefore, the lower the evaluation parameter 47 is, the more frequently the surface region of the structure 25 (closer to the work robot 22) becomes the update region 36 of the point cloud data 27, and the position / angle of the camera 21 is adjusted to adjust the measurement region. To change. As a result, even when the work area 20 covers a wide area, the point cloud data 27 forming the work area in the vicinity of the work robot 22 is ensured with high reliability.

Further, the point cloud data 27 forming the work area in the vicinity of the work robot 22 can generate the shape map 37 over the entire area of the vast work area 20 while improving the reliability. As a result, the range of utilization of the shape map 37 is expanded to a wide range of motion planning of the work robot 22 such as approaching the position of the target structure 25a from a distance. In the present embodiment, the purpose is to improve the reliability of the point cloud data 27 constituting the vicinity region of the target structure 25a to be worked by the work robot 22, but the present invention is not limited to this, and the work area 20 is not limited to this. Areas that enhance the reliability of the point cloud data 27 may be individually set according to the characteristics and the work purpose.

An operation support method for the work robot and an embodiment of the operation support program will be described with reference to the flowchart of FIG. 7 (see FIG. 6 as appropriate). The measurement signal 26 of the structure 25 (25a, 25b) measured by the camera 21, which is an example of the shape measurement sensor, is sequentially received (S11). Further, the point cloud data 27 including the three-dimensional shape of the structure 25 reproduced from the measurement signal 26 is converted from the coordinate system of the camera 21 to the coordinate system of the work area 20 (S12).

Next, the three-dimensional data 28 showing the three-dimensional shape of the work robot 22 is developed in the coordinate system of the work area 20 based on the control information 31 (S13). Then, the overlapping region 35 between the point cloud data 27 and the three-dimensional data 28 is detected in the coordinate system of the work area 20 (S14). Further, the point cloud data 27 is updated in the update area 36 (FIG. 5) excluding the overlapping area 35 (S15), the shape map 37 of the work area 20 is generated, and the point cloud data 37 is displayed on the monitor 18 (S16).

Next, the elapsed time 46 is calculated based on the camera measurement time 45 associated with each point cloud data 27 constituting the shape map 37 (S17), and each point cloud data 27 is calculated based on this elapsed time 46. Calculate reliability evaluation parameters. When this evaluation parameter is below the reference value (S18 Yes), a warning is displayed in the corresponding area of the shape map 37 of the monitor 18 and the working robot 22 is initialized (the working robot 22 is retracted from the corresponding area). Adjust the position and angle of the camera 21 (S19). Then, the flow from (S11) to (S19) is repeated until the operation of the work robot 22 is completed (S20 No: Yes END).

According to the operation support device of the work robot of at least one embodiment described above, the shape of the structure in the work area is highly reliable by detecting the overlapping area between the point cloud data of the camera and the three-dimensional data of the work robot. It becomes possible to recognize by sex.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, changes, and combinations can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention. Further, the components of the operation support device can be realized by the processor of the computer, and can be operated by the operation support program.

The operation support device 10 for the work robot described above is a control device in which processors such as a dedicated chip, FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or CPU (Central Processing Unit) are highly integrated. , Memory devices such as ROM (Read Only Memory) and RAM (Random Access Memory), external storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), display devices such as displays, mice and keyboards. It is equipped with an input device such as, and a communication I / F, and can be realized by a hardware configuration using a normal computer.

Further, the operation support device program for the work robot is provided by incorporating it into a ROM or the like in advance. Alternatively, the program is provided as a file in an installable or executable format stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD). You may try to do it.

Further, the operation support program for the work robot according to the present embodiment may be stored on a computer connected to a network such as the Internet, and may be downloaded and provided via the network. Further, the device 10 can also be configured by connecting separate modules that independently exhibit the functions of the components to each other by a network or a dedicated line and combining them.

10 (10A, 10B) ... Operation support device, 11 ... Measurement signal receiving unit (receiving unit), 12 ... Coordinate conversion unit, 15 ... Expanding unit, 16 ... Overlapping area detection unit (detection unit), 17 ... Shape map generation unit (Generation unit), 18 ... Monitor, 20 ... Work area, 21 ... Camera (shape measurement sensor), 22 ... Work robot, 25 ... Structure, 25a ... Target structure, 25b ... Peripheral structure, 26 ... Measurement signal, 27 ... Point group data, 28 ... Solid data, 30 ... Working robot control unit, 31 ... Control information, 32 ... Robot parts shape model, 33 ... Control signal, 34 ... Gripper, 35 ... Overlapping area, 36 ... Update area, 37 ... Shape map, 38 ... Link, 39 ... Servo motor, 40 ... Camera position drive unit, 41 ... Elapsed time calculation unit (calculation unit), 42 ... Reliability evaluation unit, 45 ... Camera measurement time, 46 ... Elapsed time, 47 ... evaluation parameter, 48 ... adjustment mechanism, 49 ... drive signal.

Claims (7)

A receiver that sequentially receives the measurement signals of the structure measured by the shape measurement sensor in the work area,
A coordinate conversion unit that converts point cloud data including the three-dimensional shape of the structure reproduced from the measurement signal from the coordinate system of the shape measurement sensor to the coordinate system of the work area.
A three-dimensional data showing the three-dimensional shape of the work robot is developed in the coordinate system of the work area based on the control information, and the expansion unit.
A detection unit that detects an overlapping area between the point cloud data and the three-dimensional data in the coordinate system of the work area.
An operation support device for a work robot, comprising a generation unit that updates the point cloud data in an update area other than the overlap area and generates a shape map of the work area. In the operation support device for the work robot according to claim 1.
A calculation unit that calculates the elapsed time based on the measurement time of the shape measurement sensor associated with each of the point cloud data constituting the shape map, and a calculation unit.
An operation support device for a work robot, comprising a reliability evaluation unit that evaluates the reliability of each of the point cloud data based on the elapsed time. In the operation support device for the work robot according to claim 2.
An operation support device for a work robot that performs initialization to retract the work robot from the measurement area of the shape measurement sensor based on the reliability evaluation parameter. In the operation support device for the work robot according to claim 2 or 3.
An operation support device for a work robot that changes the position of the shape measurement sensor and changes the measurement area based on the reliability evaluation parameter. The operation support device for the work robot according to any one of claims 2 to 4.
The reliability is an operation support device for a work robot, which is evaluated based on the distance between the point cloud data of the structure and the three-dimensional data of the work robot. A step in which the receiving unit sequentially receives the measurement signal of the structure measured by the camera in the work area.
In the coordinate conversion unit, the point cloud data including the three-dimensional shape of the structure reproduced from the measurement signal is converted from the coordinate system of the shape measurement sensor to the coordinate system of the work area.
In the expansion unit, the step in which the three-dimensional data showing the three-dimensional shape of the work robot is expanded in the coordinate system of the work area based on the control information, and
A step in which the detection unit detects an overlapping region between the point cloud data and the three-dimensional data in the coordinate system of the work area.
A work robot operation support method including a step in which a generation unit updates the point cloud data in an update area other than the overlapping area and generates a shape map of the work area. On the computer
Step to sequentially receive the measurement signal of the structure measured by the shape measurement sensor in the work area,
A step in which point cloud data including the three-dimensional shape of the structure reproduced from the measurement signal is converted from the coordinate system of the shape measurement sensor to the coordinate system of the work area.
A step in which three-dimensional data showing the three-dimensional shape of a work robot is developed in the coordinate system of the work area based on control information.
A step of detecting an overlapping area between the point cloud data and the three-dimensional data in the coordinate system of the work area.
An operation support program for a work robot that updates a point cloud data in an update area excluding the overlapping area and executes a step of generating a shape map of the work area.

Images