WO2022244385A1 - Robot control system, robot control method, and program - Google Patents

Abstract

One embodiment of the present invention comprises: a robot; a control device that transmits a control command to the robot; a camera that captures and obtains a captured image of at least a work environment of the robot; an image region selection device that selects at least one region of the captured image which is important for robot work, on the basis of the captured image, and sensor information acquired from a sensor that is placed on the robot and/or in the environment and that detects the state of the robot and/or the environment; and an image display device that displays the region of the captured image, which has been selected by the image region selection device.

Description

Robot control system, robot control method and program

The present invention relates to a robot control system, a robot control method, and a program that have a function of monitoring the work status of a robot.

In order to improve production efficiency and reduce labor costs, there is an increasing number of initiatives to substitute robots for tasks that were previously performed by humans, such as assembling, welding, and transporting industrial products. In these robot systems, a camera is installed at an appropriate position, and a person (operator) can operate the robot based on the image of the camera, or the robot can be automatically controlled to perform the work. many.

At this time, the appropriate position, angle of view, and number of cameras depend on the content of the work. For example, when a work area including work objects and robots is large, it is necessary to use a wide-angle camera, and the camera image becomes large. Therefore, the operator or the supervisor needs to confirm the camera image while zooming the camera image or moving the line of sight. An operator is a person who operates a robot when the robot is operated manually. A supervisor is a person who monitors the working status of the robot, whether the robot is manually operated or automatically operated. In the case of manual operation, the operator and supervisor may be the same.

In addition, in parts assembly work that includes screw tightening, etc., it is necessary to have a bird's-eye view of the work environment for recognizing situations and planning operations, and a local perspective to prevent occlusion for fine positioning. Therefore, an operator or a supervisor needs to check images while switching between a plurality of camera images.

As described above, in a robot system that uses camera images, it is necessary for the operator or monitor to move the line of sight or switch the viewpoint, which is a burden on the work. Therefore, Japanese Patent Laid-Open No. 2002-200002 describes automatically switching a suitable camera image for remote-controlling a robot based on camera image switching information stored in advance.

International Publication 2017/033359

However, although the technique described in Patent Document 1 enables switching between multiple camera images, it does not mention cropping or zooming of camera images.

In view of the above situation, there has been a demand for a method of switching between images captured by a camera while the robot is working or areas within images and presenting them to the operator or supervisor.

In order to solve the above problems, a robot control system according to one aspect of the present invention includes a robot, a control device that transmits a control command to the robot, and a camera that captures at least a working environment of the robot and acquires a captured image. , at least a part of the captured image that is important for the work of the robot, based on the captured image and sensor information obtained from a sensor that detects the state of the robot and/or the environment and/or the state of the robot and/or the environment. and an image display device for displaying the captured image region selected by the image region selection device.

According to at least one aspect of the present invention, an image or a region within the image that is important for the work of the robot captured by the camera is selected and displayed on the image display device. As a result, images can be automatically cropped and zoomed, and the burden of switching images captured by the camera and moving the line of sight of the operator and the monitor can be reduced.
Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a schematic diagram showing a configuration example of a robot control system to which the present invention is applied; FIG. 2 is a block diagram showing a hardware configuration example of an image area selection device included in the robot control system; FIG. 3 is a block diagram showing an internal configuration example of an image area selection device included in the robot control system; FIG. 4 is a flow chart showing an operation example of the image area selection device and the image display device when the camera image is a single image in the first embodiment of the present invention. FIG. 4 is a diagram showing an example of an input camera image; FIG. FIG. 4 is a diagram showing an example of a heat map for an input camera image according to the first embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of image display of the image display device in the first embodiment of the present invention; 9 is a flow chart showing an operation example of the image area selection device and the image display device when there are a plurality of camera images in the second embodiment of the present invention. FIG. 10 is a diagram showing an example (1) of an input camera image and an obtained heat map in the second embodiment of the present invention; FIG. 10 is a diagram showing an example (2) of an input camera image and an obtained heat map in the second embodiment of the present invention; FIG. 10 is a diagram showing an example (3) of an input camera image and an obtained heat map in the second embodiment of the present invention; FIG. 10 is an explanatory diagram showing an example of image display of the image display device in the second embodiment of the present invention; FIG. 10 is an explanatory diagram showing another example of image display of the image display device in the second embodiment of the present invention; 13 is a flow chart showing an operation example of an image area selection device and an image display device when automatically generating an operation command for a robot according to the third embodiment of the present invention;

Hereinafter, examples of embodiments for carrying out the present invention will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, constituent elements having substantially the same function or configuration are denoted by the same reference numerals, and overlapping descriptions are omitted.

<First Embodiment>
First, a robot control system according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 4 and 5A to 5C.

FIG. 1 is a schematic diagram showing a configuration example of a robot control system to which the present invention is applied. In the robot control system 100 shown in FIG. 1, the robot 10 is a device capable of handling objects and performing predetermined work such as assembling and transporting parts. Here, the configuration of the robot 10 does not matter, and it may be a single robot arm, or may include a moving device such as a crawler or wheels.

The robot control device 20 outputs control commands to the robot 10 based on motion commands such as joint angles and forces (torques) of the robot 10 input to the robot control device 20 to control the motion of the robot 10 . is. The control command is, for example, a signal indicating a current value, a voltage value, or the like for an actuator (such as a motor) provided at a joint of the robot 10, the end effector 11, or the like. When the robot 10 receives a control command from the robot controller 20, the built-in drive circuit supplies a drive signal to the corresponding actuator.

The robot control device 20 may be configured to automatically control the motion of the robot 10 based on the camera image captured by the camera 3. A configuration in which the robot control device 20 automatically controls the motion of the robot 10 will be described in a third embodiment.

The camera 3 is an imaging device for imaging the working environment of the robot 10 and the surrounding environment. The number of cameras 3 may be one or a plurality of them as shown in FIG. In FIG. 1, three cameras 3a, 3b, and 3c are installed as the cameras 3. In FIG. The camera 3a is attached to the robot arm, and the cameras 3b and 3c are installed around the robot 10 (for example, walls of a working room or a building). Here, the work environment corresponds to the movable area of the robot 10, that is, the range in which the robot 10 can move (work area). The surrounding environment corresponds to the surrounding area outside the movable area of the robot 10 . The camera 3 is assumed to be a network camera without attitude change function and hardware zoom function.

In this embodiment, the case of a single camera image will be described, and in the second embodiment, the case of multiple camera images will be described.

The image region selection device 30 selects the work of the robot 10 based on the information obtained from the camera image captured by the camera 3 and the sensors arranged in the robot 10 and/or the environment (hereinafter referred to as "sensor information"). It is a device that selects important images and areas within those images. Here, the type of sensor information of the robot 10 and the environment does not matter. For example, the sensor information may be a current value of a motor provided in a joint of the robot 10, an output signal of a tactile sensor or an inertial sensor externally attached to the robot 10, or the like. Furthermore, the sensor information may be a temperature sensor that measures the work environment, an output signal from a line-of-sight sensor (not shown) attached to the operator of the robot 10, or the like. In this way, each sensor detects the state of the robot 10 and/or the state of the environment, and outputs a detection signal according to the content of detection.

The image display device 40 is a device that selectively displays the image selected by the image region selection device 30 and the region within the image. The robot operating device 50 is provided with buttons, a joystick, and the like corresponding to the motion of the robot 10 , and is a device that receives input from the operator and transmits an operation command corresponding to the content of the input to the robot control device 20 .

[Hardware configuration of each device]
Next, the hardware configuration of the image area selection device 30 included in the robot control system 100 will be described with reference to FIG. Here, a hardware configuration example of a computer included in the image region selection device 30 will be described.

FIG. 2 is a block diagram showing a hardware configuration example of a computer included in the image area selection device 30. As shown in FIG. The illustrated computer 60 is an example of hardware that constitutes a computer used in the robot control device 20 and the image area selection device 30 . A personal computer, for example, can be used as the computer 60 .

The computer 60 includes a CPU (Central Processing Unit) 61, a ROM (Read Only Memory) 62, and a RAM (Random Access Memory) 63 connected to a bus 64 respectively. Computer 60 further comprises non-volatile storage 66 , input/output interface 67 and network interface 68 .

The CPU 61 reads the program code of the software that implements the functions of the image area selection device 30 according to this embodiment from the ROM 62, loads the program into the RAM 63, and executes it. In the RAM 63, variables, parameters, etc. generated during the arithmetic processing of the CPU 61 are temporarily written. Variables and parameters written in the RAM 63 are appropriately read by the CPU 61 . Although the CPU 61 is used as the arithmetic processing unit, other processors such as MPU (Micro Processing Unit) may be used.

The non-volatile storage 66 is an example of a recording medium, and can store data used by programs and data obtained by executing programs. For example, the nonvolatile storage 66 stores learning data, learning models, etc., which will be described later. Also, an OS (Operating System) and a program executed by the CPU 61 may be recorded in the nonvolatile storage 66 . As the nonvolatile storage 66, a semiconductor memory, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a disk device using magnetism or light, or the like is used.

The input/output interface 67 is an interface that communicates signals and data with each sensor and each actuator provided in the robot control system 100 . The input/output interface 67 may also serve as an A/D (Analog/digital) converter and/or a D/A converter (not shown) that processes an input signal or an output signal. The sensor information herein includes information obtained from each actuator as well as from each sensor.

For the network interface 68, for example, a NIC (Network Interface Card), modem, or the like is used. The network interface 68 is configured to be capable of transmitting and receiving various data to and from an external device via a communication network such as a LAN or the Internet to which terminals are connected, a dedicated line, or the like.

[Internal configuration of image area selection device]
FIG. 3 is a block diagram showing an internal configuration example of the image region selection device 30. As shown in FIG. The image region selection device 30 includes a learning data storage unit 31 , a learning unit 32 , a learning model storage unit 33 and an inference unit 34 .

The learning data storage unit 31 stores learning data used for learning the learning model 33a. The learning data includes at least camera images of the work environment of the robot 10 and the surrounding environment acquired in time series during the work of the robot 10, and sensor information of the robot 10 and the environment.

The learning unit 32 uses learning data stored in the learning data storage unit 31 to learn predicted values of sensor information of the robot 10 and the environment, and heat maps (important areas of camera images, importance). Let the model 33a learn. For example, the learning unit 32 adjusts the model parameters of the learning model 33a by learning the learning model 33a through machine learning. The learning model 33a can be configured using a neural network as an example. The learning method of the learning model 33a is not limited to machine learning based on deep learning using a neural network, and other learning methods may be used.

The learning model storage unit 33 stores the learning model 33a and its model parameters. For example, when a neural network is used for the learning model 33a, the model parameters are weights such as the degree of connectivity between neurons forming the neural network and the firing threshold of neurons. The learning model storage unit 33 is realized by the nonvolatile storage 66 as an example. The learning model 33a stored in the learning model storage unit 33 is a trained model (inference program) in which learning results (model parameters) are reflected.

The inference unit 34 uses the learning model 33a stored in the learning model storage unit 33 to infer (predict) the input camera image and the sensor information of the robot 10 and the environment, and outputs their predicted values. do. The inference unit 34 also includes a heat map generation unit 331 that generates a heat map, and uses the learning model 33a to infer a heat map (important region, importance) for the input camera image. Details of the heat map and the heat map generator 331 will be described later. The result of inference by the inference unit 34 is output to the robot control device 20 and the image display device 40 . The inference unit 34 processes (post-processes) the camera image based on the inference result, and outputs data necessary for displaying the image on the image display device 40 . As the processing of the camera image, for example, cutting out a specific area and enlarging it can be mentioned.

Next, in the robot control system 100, an example of how the image area selection device 30 selects an important area in the camera image for the robot 10 to perform a task (hereinafter referred to as "important area") will be described. . The method of selecting important regions in the camera image is divided into a teaching phase and an operating phase. For example, by preparing a learning mode and an operation mode in the robot control system 100 and selecting either the learning mode or the operation mode displayed on a menu screen (not shown) with the robot operation device 50, the teaching phase or the operation phase can be started. Transition.

[Teaching phase of robot control system]
In the teaching phase, the robot 10 is used to perform the work with the image area selection function of the image area selection device 30 disabled. The image area selection device 30 saves the camera image acquired during the work and the sensor information of the robot 10 and the environment as learning data in the learning data storage unit 31 . Learning in the learning unit 32 is performed using the learning data at this time as teacher data. The operator may control the robot 10 using the robot operating device 50 , or the robot 10 may automatically reproduce the previously planned motion of the robot 10 .

Next, in the learning unit 32 of the image region selection device 30, when the robot 10 performs a certain task, the next time (for example, The model parameters of the learning model 33 a that predicts the camera image after time t+1) and the sensor information of the robot 10 and the environment are learned and stored in the learning model storage unit 33 . The learning of the learning model 33a is performed for each work type. Furthermore, the accuracy of learning increases by performing learning multiple times for one type of work.

As described above, the learning model 33a has a heat map generation unit 331 that generates a heat map indicating areas within the camera image, particularly image areas (important areas) necessary for the prediction. The learning model 33a is configured such that the heat map generation unit 331 optimizes the heat map value so that the image region necessary for prediction is large or small so that the prediction error for each data is reduced in the process of learning. It has become. Each data is a camera image, sensor information of the robot 10 and the environment. Then, the learning unit 32 is configured so as to set an area in which the values of the heat map generated by the heat map generating unit 331 of the learning model 33a are equal to or greater than the set threshold value as the important area. The important area in this embodiment is assumed to be at least part of the camera image, but may be the entire image.

The learning model 33a is composed of a camera image at time t+1, sensor information of the robot 10 and the environment, and a camera image corresponding to the time t+1 stored in the learning model storage unit 33, and sensor information of the robot 10 and the environment. and calculate the error of each data (corresponding to the prediction error). In this embodiment, the value of this heat map is regarded as the "importance" of the region in the camera image. Here, the image region is not limited to a region having a certain area. A region of an image can be a point and a heatmap value can be a single value at that point.

An image or a region within the image that is important for the work of the robot 10 is searched according to a predetermined region search algorithm. The shape of the area is assumed to be circular, but may be elliptical or rectangular. As an example, the following area search algorithm is conceivable. First, the learning model 33a detects a work object in a camera image, and sets a position (x coordinate, y coordinate) as a search reference in the image based on the work type, the work object, and the like. The learning model 33a configures a temporary area with a plurality of pixels contained within a certain radius centered on the reference position, and calculates a prediction error using the image of the temporary area. Next, the learning model 33a changes the range of the temporary region (for example, reduces the range) and calculates the prediction error.

Then, the learning model 33a compares the prediction error when using the previous provisional region and the prediction error when using the current provisional region, and determines whether the prediction error decreases from the previous time. If the prediction error becomes smaller, the learning model 33a further reduces the range of the temporary region, calculates and compares the prediction error, and repeats these processes. When the prediction error gradually decreases and then increases, the temporary area having the range and the reference position immediately before the prediction error starts to increase is considered to be an important area for the work of the robot 10 .

Furthermore, by calculating and comparing prediction errors by changing not only the range of the temporary area but also the reference position in the image of the temporary area, it is possible to specify an area that can be estimated to be important for the work of the robot 10 . The heat map generator 331 of the learning model 33a calculates the prediction error for each pixel in the image or for the pixel representing the region in the process of setting the range of the region and calculating and comparing the prediction error. Generate a heatmap by giving heatmap values. As described above, a region composed of pixels whose heat map values are greater than or equal to a set threshold is specified as an important region. Note that depending on the type of work (for example, work that requires a bird's-eye view), calculation and comparison of prediction errors are performed while enlarging the range of the temporary area, and areas important to the work of the robot 10 are specified.

[Robot control system operation phase]
Next, operation phases of the robot control system 100 will be described with reference to FIG. FIG. 4 is a flow chart showing an operation example of the inference section 34 of the image area selection device 30 and the image display device 40 when the camera image is a single image. First, the robot control device 20 receives an operation command for the robot 10 input by the operator from the robot operation device 50, and generates a control command for each actuator for controlling the robot 10 based on the operation command. The motion command is, for example, the joint angle of the robot 10, the posture (position) and force (torque) of the end effector 11, and the like. The robot 10 receives a control command from the robot control device 20 and starts an operation (work) (S1).

Next, the operator or supervisor confirms the camera image displayed on the screen of the image display device 40 and determines whether the work of the robot 10 has been completed (S2). For example, an icon of a work completion button 74 (see FIG. 5C described later) is displayed in the camera image displayed on the image display device 40 . When the operator or supervisor determines that the work is completed, the robot operation device 50 is used to click the icon of the work completion button 74 . When the inference unit 34 of the image area selection device 30 detects that the work completion button 74 has been operated, it determines that the work has been completed. A mechanical work end button may be arranged on the robot operating device 50 .

When the operator or supervisor determines that the work of the robot 10 is completed (YES in S2), the work of the robot 10 is finished after operating the work completion button 74 (S7).

On the other hand, if the operator or supervisor determines that the work of the robot 10 has not been completed (NO in S2), the inference unit 34 acquires the camera image and the sensor information of the robot 10 and the environment to create a learning model. 33a (S3) to obtain an output (inference result) corresponding to the input.

Next, the inference unit 34 acquires the important region of the camera image as an output in response to the input of the camera image to the learning model 33a in step S3, the sensor information of the robot 10 and the environment (S4). The important areas of the camera image correspond to areas with high values (importance) in the heatmap. In addition, the inference unit 34 acquires predicted values of the camera image and the sensor information of the robot 10 and the environment as an output corresponding to the input of each data to the learning model 33a in step S3 (S5). This predicted value is used when the robot 10 automatically performs the work in the third embodiment.

Next, the image area selection device 30 transmits information (for example, position, range) of the important area of the camera image inferred by the inference unit 34 to the image display device 40 . As a result, the image region selection device 30 displays on the image display device 40 an image including the important region in the camera image obtained by the inference unit 34 in step S4 (S6). As a result, the image display device 40 enlarges and displays the important area in the camera image on the screen. After the processing of steps S5 and S6, the process returns to the judgment processing of step S2, and if it is not judged that the work is completed, the camera image and the sensor information of the robot 10 and the environment are input to the learning model 33a. In this way, the processing of steps S2 to S6 is repeated until it is determined that the work is completed, that is, from the start of the work to the end of the work. FIG. 5C shows an image display example of the image display device 40 .

[Image display example]
5A to 5C show examples of heat maps and image displays for input camera images in the first embodiment. 5A is an input camera image, FIG. 5B is a heat map, and FIG. 5C is an image display example of the image display device 40. FIG. In the example of the heat map shown in FIG. 5B, as an example, areas with high heat map values, i.e., areas with high importance, are expressed darkly, and areas with small heat map values, i.e., areas with low importance, are expressed lightly.

When the heat map 72 shown in FIG. 5B is obtained for the input camera image 71 shown in FIG. 5A, the image display device 40 displays the image of the area (specific area Ai) having a large value in the heat map 72 as shown in FIG. 5C. to display. FIG. 5C shows an example of an image 73 in which an area corresponding to the specific area Ai of the heat map 72 in the input camera image 71 is enlarged and displayed. Specifically, in the image 73, the local area including the work object 12 and the end effector 11 of the image 71, which is the input camera image, is enlarged.

As described above, the robot control system (robot control system 100) according to the first embodiment includes a robot (robot 10) that performs a task and a control device (robot control device) that transmits control commands to the robot based on motion commands. 20), a camera (camera 3) that captures at least the work environment of the robot and acquires a captured image, the captured image, the state of the robot and/or the robot placed in the environment, and/or the state of the environment. an image region selection device (image region selection device 30) that selects at least a partial region of the captured image that is important for the work of the robot, based on the sensor information acquired from the sensor, and the image region selection device that is selected by the image region selection device. and an image display device (image display device 40) for displaying the captured image area.

The robot control system 100 according to the first embodiment configured as described above sequentially acquires the important regions of the image captured by the camera 3, and displays the image or the region within the image on the image display device 40. By updating, images important for work and areas within the images are presented to the operator or supervisor according to the work situation. That is, the robot control system 100 uses the image area selection device 30 to automatically cut out an image that is important for the work of the robot 10 captured by the camera 3 or an area within the image, and enlarge and display it on the image display device 40 . As a result, the robot control system 100 can reduce the load on the operator or the observer due to switching between images captured by the camera 3 or areas within the images and movement of the line of sight. For example, since the work area is wide, an image from a viewpoint that overlooks the work environment and an image from a viewpoint that requires detailed information can be automatically switched and displayed on the image display device 40 .

Further, in the robot control system (robot control system 100) according to the first embodiment, the image area selection device (image area selection device 30) selects the following from the captured image captured by the camera at the current time and the sensor information: A learning model (learning model 33a) that has been trained to predict captured images and sensor information after that time, and a heat map generation unit (heat map generator 331). Then, the learning model is optimized so that the values in the region necessary for prediction become larger or smaller so that the prediction error is reduced in the process of learning. Let the area be an important area (specific area Ai).

Further, in the robot control system (robot control system 100) according to the first embodiment, a model storage unit that stores a learning model (learning model 33a) for selecting an important region (specific region Ai) in a captured image; a learning unit for learning a learning model; a learning data storage unit for storing time-series learning data including at least captured images and sensor information obtained during the robot's work and used for learning the learning model; and a model storage. an inference unit that infers important regions in the captured image using the learning model stored in the unit.

Note that there may be a plurality of important areas for one camera image. For example, when a plurality of important areas exist within a camera image, the camera image may be divided and an image may be displayed for each important area. At this time, if there are a large number of important areas in the camera image, it is preferable to obtain the degree of importance for each important area in step S4, and to display the image of the important area with the highest degree of importance first.

<Second embodiment>
Next, as a second embodiment of the present invention, a robot control system in which there are a plurality of camera images will be described with reference to FIGS. 6 to 9. FIG. The basic configuration of the robot control system according to the second embodiment is the same as the robot control system 100 shown in FIG.

FIG. 6 is a flow chart showing an operation example of the inference section 34 of the image area selection device 30 and the image display device 40 when there are a plurality of camera images in the second embodiment. The second embodiment is different from the first embodiment in that there are a plurality of camera images acquired by the camera 3, and the flowchart of FIG. 6 is different from that of FIG. 4 shown in the first embodiment. The different parts are steps S14 and S16-S17. Since the processes of steps S11 to S13, S15 and S18 are the same as the processes of steps S1 to S3, S5 and S7 in FIG. 4, detailed description thereof will be omitted.

In FIG. 6, when the operator or supervisor determines that the work of the robot 10 has not been completed (NO in S12), the inference unit 34 of the image region selection device 30 selects a plurality of camera images, the robot 10 and the environment. sensor information is acquired and input to the learning model 33a (S13), and an output (inference result) corresponding to the input is obtained.

Next, the inference unit 34 outputs important regions and importance (heat map values) for each of the plurality of camera images as outputs for the input of the plurality of camera images, the sensor information of the robot 10 and the environment to the learning model 33a in step S13. (S14). In addition, the inference unit 34 acquires predicted values of a plurality of camera images, sensor information of the robot 10 and the environment as an output corresponding to the input of each data to the learning model 33a in step S13 (S15).

Next, the inference unit 34 compares the degrees of importance acquired in step S14 among a plurality of camera images, and selects a camera image or demand area to be displayed (S16). In this example, two or more camera images are selected in descending order of importance.

The image region selection device 30 then transmits the camera image or the important region selected by the inference unit 34 to the image display device 40 . Thereby, the image region selection device 30 displays the camera image or the important region obtained by the inference unit 34 in step S14 on the image display device 40 (S17).

After the processing of steps S15 and S17, the process returns to the determination processing of step S12, and if the work is not completed (NO in S12), the processing of steps S13 to S17 is repeated. 7A to 7C show image display examples of the image display device 40. FIG.

7A to 7C show examples of input camera images and obtained heat maps in the second embodiment. For example, FIG. 7A is an input camera image (image 81) of camera 3a, FIG. 7B is an input camera image (image 82) of a camera such as camera 3c that can overlook the entire robot 10, and FIG. 7C is an input camera image of camera 3b. (image 83). Images 81 to 83 in FIGS. 7A to 7C respectively show specific areas Ai1 to Ai3 having relatively higher heat map values than other pixels.

[Example of image display]
FIG. 8 shows an example of image display of the image display device 40 in the second embodiment. In FIG. 8 , an image 91 corresponds to the input camera image 81 , an image 92 corresponds to the input camera image 82 , and an image 93 corresponds to the input camera image 83 .

As shown in FIGS. 7A to 7C, when heat maps each including specific areas Ai1 to Ai3 are obtained for the plurality of input camera images 81 to 83 in FIG. Priority is given to the camera image of the area with a high degree of importance with a large value of . FIG. 8 shows an example in which camera images to be displayed are arranged in order (predetermined positions) according to the degree of importance of the camera images. Here, on the display screen 90, an image is displayed in a large size according to the degree of importance of the camera image, and other small images are displayed in a reduced size. By determining the arrangement of the plurality of images acquired by the camera 3 according to the degree of importance in this manner, it becomes clear to the operator or the supervisor which image should be checked with priority. Furthermore, since the areas (sizes) of the images differ depending on the level of priority, it becomes clearer for the operator or supervisor which images should be checked with priority.

For example, in FIGS. 7A to 7C, the heat map values in the specific areas Ai1 to Ai3 are larger in the order of Ai1>Ai2>Ai1. Therefore, on the display screen 90 shown in FIG. 8, an image 91 with the largest heat map value is displayed in the left and center areas, and an image 92 with the second heat map value and an image 93 with the third heat map value are displayed. , are reduced at the same reduction ratio and displayed vertically in the area on the right. It goes without saying that the arrangement of the image with the largest heat map value and the other images is not limited to the arrangement shown in FIG.

Here, regardless of the image display method (image size, shape, arrangement, etc.) in the image display device 40, the area of the image to be displayed is determined according to the value (importance) of the heat map as shown in FIG. You can change it.

[Another example of image display]
FIG. 9 shows another example of image display of the image display device 40 in the second embodiment. The example shown in FIG. 9 is an example in which the area of the camera image to be displayed is changed according to the importance of the camera image.

Images 91A to 93A corresponding to the images 91 to 93 shown in FIG. 8 are displayed on the display screen 90A shown in FIG. The relative positional relationship of the images 91A-93A and the ratio of the long side to the short side (aspect ratio) of each image are the same as those of the images 91-93. However, the relation of the areas (sizes) of the images 91A-93A is different from that of the images 91-93. That is, in FIG. 9, the areas of the image 92A and the image 93A are different. In this way, images may be displayed in larger sizes in descending order of heat map value (higher importance). By determining the areas of the plurality of images acquired by the camera 3 according to the degree of importance in this way, it becomes clear to the operator or the supervisor which image should be checked with priority.

It should be noted that the display screen 90A has blank spaces below the image 91A and to the right of the image 93A. In order to effectively use the display area of the display screen 90A, the size, shape, and arrangement of the images 91A to 93A can be appropriately changed within the range of the rule that images are displayed larger in order of importance.

The robot control system 100 according to the second embodiment configured as described above successively acquires the important regions and importance of the images that are important for the work of the robot 10 when there are a plurality of images captured by the camera 3. Then, by updating the display of images on the image display device 40 based on the degree of importance of each image, images important to the work are presented to the operator or supervisor according to the work situation. That is, the robot control system 100 uses the image region selection device 30 to compare the degrees of importance of a plurality of images captured by the camera 3, automatically selects an image based on the degree of importance, and displays it on the image display device 40. . As a result, the robot control system 100 according to the present embodiment can reduce the load on the operator and the monitor due to switching between a plurality of images captured by the plurality of cameras 3 and movement of the line of sight. For example, a plurality of images, such as an image from a bird's-eye view of the working environment and an image from a local viewpoint to prevent occlusion for fine positioning, can be automatically switched and displayed on the image display device 40 .

The configuration for cutting and enlarging the image area of the camera image in the first embodiment may be applied to the robot control system 100 according to the second embodiment that switches between a plurality of camera images. With this configuration, the effects of the first embodiment can be obtained in addition to the effects of the second embodiment.

<Third Embodiment>
Next, as a third embodiment of the present invention, a robot control system in which an operation command for the robot 10 is automatically generated and the robot 10 operates autonomously in parallel with displaying an image on the image display device 40. Description will be made with reference to FIG. The basic configuration of the robot control system according to the third embodiment is the same as the robot control system 100 shown in FIG.

FIG. 10 is a flow chart showing an operation example of the inference section 34 of the image area selection device 30 and the image display device 40 when automatically generating an action command for the robot 10 in the third embodiment. The third embodiment differs from the first embodiment in that an operation command for the robot 10 is automatically generated. The difference is that step S27 is added. Since the processing of steps S21 to S26 and S28 is the same as the processing of steps S1 to S7 in FIG. 4, detailed description thereof will be omitted. In the present embodiment, the weight area (and the degree of importance) of the acquired camera image and the predicted value are used to generate the motion command for the robot 10 .

The image region selection device 30 determines the motion of the robot 10 based on the important region (importance) of the camera image output from the learning model 33a by the inference unit 34, the camera image, and the predicted values of the sensor information of the robot 10 and the environment. The robot 10 is controlled by sending commands to the robot control device 20 . At this time, the input from the robot operating device 50 may be accepted so that the operator can operate the robot 10, or the input from the robot operating device 50 may not be accepted and the robot 10 may be operated completely automatically. The processing of step S27 is as follows.

After completing the process of step S25, the inference unit 34 inputs to the robot control device 20 the predicted value regarding the motion of the robot 10 among the predicted values of the camera image, the robot 10, and the sensor information acquired in step S25 (S27 ). The robot control device 20 outputs a control command to the robot 10 using the predicted value input from the image area selection device 30 as an action command, thereby controlling the action (work) of the robot 10 . Here, the predicted value regarding the motion of the robot 10 corresponds to the motion command of the robot 10, and includes, for example, the joint angle of the robot 10, the posture (position) and force (torque) of the end effector 11, and the like.

After the processing of steps S26 and S27, the process returns to the judgment processing of step S22, and if the work is not completed (NO in S22), the processing of steps S23 to S27 is repeated.

The robot control system 100 according to the third embodiment configured as described above updates the display of an image or an area within the image on the image display device 40 while the robot 10 automatically performs a task. , to present an image important to the work and a region within the image to the operator or supervisor according to the work situation. As a result, the robot control system 100 according to the present embodiment, as in the case of the first embodiment, allows the operator or the observer to switch between the image captured by the camera 3 or the area within the image and move the line of sight. In addition to being able to reduce the load caused by the robot 10, the robot 10 can be made to automatically perform the work.

Note that the configuration for switching between a plurality of camera images in the second embodiment may be applied to the robot control system 100 according to the third embodiment that automatically generates motion commands for the robot 10 . With this configuration, the effects of the second embodiment can be obtained in addition to the effects of the third embodiment. In other words, the robot control system 100 according to the present embodiment can reduce the load on the operator or the observer due to switching between a plurality of images captured by the plurality of cameras 3 and movement of the line of sight, and in addition, the robot 10 can be automatically operated. can be implemented.

Also, in the third embodiment, since the robot 10 automatically generates motion commands, the robot operating device 50 of the robot control system 100 can be deleted. In addition, in the third embodiment, since the operator does not operate the robot operation device 50, confirmation of the work status using the image display device 40 for inputting operation commands may be unnecessary. Therefore, the image display device 40 of the robot control system 100 may be deleted. However, the image display device 40 may remain in the robot control system 100 in order for the observer to check the work status of the automatic operation of the robot 10 .

<Modification>
It should be noted that the present invention is not limited to the first to third embodiments described above, and various other applications and modifications can be made without departing from the gist of the present invention described in the claims. is of course. For example, each of the above-described embodiments is a detailed and specific description of the configuration of the robot control system and the image area selection device in order to explain the present invention in an easy-to-understand manner. Not limited. Also, it is possible to replace part of the configuration of one embodiment with the constituent elements of another embodiment. It is also possible to add components of other embodiments to the configuration of one embodiment. Moreover, it is also possible to add, replace, or delete other components for a part of the configuration of each embodiment.

In addition, the image area selection device 30 and each camera 3, or the robot control device 20 and each camera 3 are configured to be able to communicate with each other, and each camera 3 is provided with a posture change function and a hardware zoom function. can be For example, when a person operates the robot 10 in the teaching phase, the robot operation device 50 outputs commands such as camera posture and zoom to the camera 3 via the image area selection device 30 or the robot control device 20 . As a result, the operator can change the posture and zoom the camera 3 for the purpose of displaying an image that facilitates the operation of the robot 10 . The camera image at this time is used as learning data. Further, in the third embodiment, when automatically generating an operation command for the robot 10, the image obtained by the camera 3 is not limited to being post-processed (clipping and enlarging the area) and displayed. It is possible to employ a configuration in which the image area selection device 30 directly gives a zoom command to the camera 3 to acquire an image depending on the situation.

In addition, each of the above configurations, functions, processing units, etc. may be realized by hardware, for example, by designing a part or all of them with an integrated circuit. As hardware, a broadly defined processor device such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may be used.

Also, each component of the image region selection device 30 according to each embodiment described above may be implemented in the robot control device 20 . Further, the processing performed by a certain processing unit of the image region selection device 30 may be implemented by one piece of hardware, or may be implemented by distributed processing by a plurality of pieces of hardware.

In addition, in the flowcharts shown in FIGS. 4, 6, and 10, multiple processes may be executed in parallel or the order of the processes may be changed as long as the processing results are not affected.

DESCRIPTION OF SYMBOLS 10... Robot 20... Robot control apparatus 3, 3a, 3b, 3c... Cameras 30... Image area selection apparatus 40... Image display apparatus 50... Robot operation apparatus 31... Learning data storage part 32... Learning Unit 33 Learning model storage unit 33a Learning model 331 Heat map generation unit 34 Inference unit 60 Calculator 61 CPU 71 Input camera image 72 Heat map 73 Image 74 ... work completion button, 81, 82, 83 ... image (input camera image and heat map), 90, 90A ... display screen, 91 to 93, 91A to 93A ... image, 100 ... robot control system

Claims (12)

1. a robot that does the work,
   a control device that transmits a control command to the robot based on the motion command;
   a camera that captures at least the work environment of the robot and acquires a captured image;
   Based on the captured image and sensor information obtained from a sensor that detects the state of the robot and/or the environment and/or the state of the environment, the captured image that is important for the work of the robot is generated. an image region selection device for selecting at least a partial region;
   an image display device that displays the region of the captured image selected by the image region selection device;
   robot control system.
2. The image area selection device comprises:
   a learning model trained to predict the captured image and the sensor information after the next time from the captured image captured by the camera at the current time and the sensor information;
   a heat map generation unit that generates a heat map indicating an area in the captured image inside the learning model,
   The learning model is optimized so that the values in the regions necessary for prediction increase or decrease so that the prediction error is reduced in the process of learning. 2. The robot control system according to claim 1, wherein is the important area.
3. The image area selection device comprises:
   a model storage unit that stores the learning model for selecting the important region in the captured image;
   a learning unit that learns the learning model;
   a learning data storage unit that stores time-series learning data that includes at least the captured image and the sensor information obtained during the operation of the robot and that is used for learning the learning model;
   The robot control system according to claim 2, further comprising an inference section that infers the important region in the captured image using the learning model stored in the model storage section.
4. The inference unit compares the importance levels of the captured images output by the inference unit of the image area selection device among a plurality of the captured images based on the values of the heat map for each of the captured images, and 4. The robot control system according to claim 3, wherein an arrangement of said captured images displayed on at least said image display device is changed depending on the degree of movement.
5. The inference unit compares the importance levels of the captured images output by the inference unit of the image area selection device among a plurality of the captured images based on the values of the heat map for each of the captured images, and 4. The robot control system according to claim 3, wherein the area of the captured image displayed on the image display device is changed according to the degree.
6. 2. The robot control system according to claim 1, wherein the image display device displays on a screen an image obtained by cutting out the region of the captured image selected by the image region selection device.
7. 2. The robot control system according to claim 1, further comprising a robot operation device that transmits the action command to the control device according to an operation by an operator.
8. a robot that does the work,
   a control device that transmits a control command to the robot based on the motion command;
   a camera that captures at least the work environment of the robot and acquires a captured image;
   Based on the captured image and sensor information obtained from a sensor that detects the state of the robot and/or the environment and/or the state of the environment, the captured image that is important for the work of the robot is generated. an image region selection device that selects at least a partial region;
   The image area selection device comprises:
   a learning model trained to predict the captured image and the sensor information after the next time from the captured image captured by the camera at the current time and the sensor information;
   a heat map generation unit that generates a heat map indicating an area in the captured image inside the learning model,
   The learning model is optimized so that the value in the region necessary for prediction is large so that the prediction error is reduced in the process of learning. year,
   A robot control system, wherein the image area selection device outputs a predicted value related to the motion of the robot, out of predicted values of the captured image and the sensor information based on the learning model, to the control device as the motion command.
9. a control device that transmits a control command to the robot based on the motion command;
   a camera that captures at least the work environment of the robot and acquires a captured image;
   Based on the captured image and sensor information obtained from a sensor that detects the state of the robot and/or the environment and/or the state of the environment, the captured image that is important for the work of the robot is generated. an image region selection device for selecting at least a partial region;
   an image display device that displays the region of the captured image selected by the image region selection device;
   robot control system.
10. a control device that transmits a control command to the robot based on the motion command;
    a camera that captures at least the work environment of the robot and acquires a captured image;
    Based on the captured image and sensor information obtained from a sensor that detects the state of the robot and/or the environment and/or the state of the environment, the captured image that is important for the work of the robot is generated. an image region selection device that selects at least a partial region;
    The image area selection device comprises:
    a learning model trained to predict the captured image and the sensor information after the next time from the captured image captured by the camera at the current time and the sensor information;
    a heat map generation unit that generates a heat map indicating an area in the captured image inside the learning model,
    The learning model is optimized so that the value in the region necessary for prediction is large so that the prediction error is reduced in the process of learning. year,
    A robot control system, wherein the image area selection device outputs a predicted value related to the motion of the robot, out of predicted values of the captured image and the sensor information based on the learning model, to the control device as the motion command.
11. A robot control method by a robot control system comprising a control device for transmitting a control command to a robot based on a motion command and an image area selection device,
    The image region selection device,
    Based on at least a captured image acquired by a camera that captures the work environment of the robot and sensor information acquired from a sensor that detects the state of the robot and/or the robot placed in the environment and/or the state of the environment. a process of selecting at least a partial area of the captured image that is important for the work of the robot;
    A robot control method for executing a process of outputting the selected area of the captured image to an image display device.
12. A computer provided in the image area selection device of a robot control system composed of a control device that transmits a control command to the robot based on an operation command and an image region selection device,
    Based on at least a captured image acquired by a camera that captures the work environment of the robot and sensor information acquired from a sensor that detects the state of the robot and/or the robot placed in the environment and/or the state of the environment. selecting at least a partial region of the captured image that is important for the work of the robot;
    A program for executing a procedure of outputting the selected area of the captured image to an image display device.

Images