WO2020213194A1

WO2020213194A1 - Display control system and display control method

Info

Publication number: WO2020213194A1
Application number: PCT/JP2019/040676
Authority: WO
Inventors: 吉田　修一; 剛大濱; 勁峰今西; 良一今中
Original assignee: 日本金銭機械株式会社
Priority date: 2019-04-15
Filing date: 2019-10-16
Publication date: 2020-10-22
Also published as: JP6647640B1; JP2020172011A

Abstract

The present invention realizes a technique with which it is possible to improve safety of work while maintaining high work efficiency by suitably displaying a safe region when a robot arm and a person are performing a shared operation. In this display control system (1000), a prediction process is performed using a learned model in which an image captured by an imaging unit installed above a robot arm (Rbt_arm), such as an image in which it is possible to distinguish the position and state of the robot arm as well as the position and state of a worker, and data that specifies a safe region when the image is acquired are learned as training data. Moreover, in the display control system (1000), the prediction process carried out through the learned model is performed using an image captured under the same conditions as those in learning, whereby a safe region from when the inputted image is captured is predicted, and the safe region is projected onto a projection surface by a projection unit.

Description

Display control system and display control method

The present invention relates to a technique for detecting danger when a human enters the robot work area and a technique for displaying a danger area (or safety area).

In recent years, the situation where robots and humans work together is increasing. Along with this, the possibility of collision between humans and robots increases. For example, it is difficult for a human to predict the movement of a robot arm having a robot arm having a plurality of joints because the movement of the robot is complicated. As a result, the sudden movement of the robot arm may cause the robot arm to come into contact with a human and cause a serious accident.

In order to ensure safety during work by the robot arm, for example, in the technique disclosed in Patent Document 1, a collision is detected by a force sensor in the joint of the robot arm, and when the collision is detected, the robot arm is stopped.

Japanese Unexamined Patent Publication No. 2006-21287

However, as in the conventional technology, the safety device may be activated after the robot arm and a human contact (collision), or the possibility of collision may be predicted and the robot arm may be controlled to move slowly. , There is a problem that work efficiency is lowered.

Therefore, the present invention is a technique for improving work safety while ensuring high work efficiency by appropriately displaying a safety area (or a danger area) when a robot arm and a human perform joint work. The purpose is to realize.

In order to solve the above problems, the first invention is a safety area which is a region where it is determined that it is safe even if a movable object exists in a space where a robot arm and a movable object may coexist. Is a display control system for displaying a moving object on a projection surface that can be recognized in space, and includes an imaging unit, a prediction processing unit, and a projection unit.

The imaging unit is installed at a position above the robot arm (for example, a position where the space within the movable range of the robot arm can be photographed).

The prediction processing unit is (1) an image captured by the imaging unit from a position above the robot arm in space, and is an image captured when the robot arm is in a predetermined state, or the robot arm is Learning processing is performed using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for specifying a safety area when the robot arm is in the predetermined state as teacher data. Prediction processing is executed using the trained model acquired by execution. Further, at the time of prediction processing, the prediction processing unit executes prediction processing using the trained model on the prediction processing image which is an image captured from a position above the robot arm by the imaging unit. , The safety area in the space when the image for prediction processing is acquired is predicted, the predicted safety area is acquired as the prediction safety area, and the projected image data is generated based on the prediction safety area.

The projection unit projects the projected image formed by the projected image data onto the projection surface.

In this display control system, images (for example, frame images) captured by an imaging unit installed above the robot arm, for example, (1) the position, state, and movable object (for example, a worker) of the robot arm. An image (for example, a frame image) capable of determining the position and state of the robot arm, or control data for controlling the robot arm so that the robot arm is in a predetermined state, and (2) the robot arm is in the predetermined state. As training data (teacher data), the data that identifies the safety area at the time of is used as training data, and prediction processing is performed using the trained trained model. Then, in this display control system, when the input image is captured by performing the prediction processing by the trained model using the image (for example, the frame image) captured in the same state as when it was trained. The safety area (safe area on the projection plane) can be predicted (specified). Then, in this display control system, the predicted (specified) safety area is projected onto the projection surface (for example, the floor surface) by the projection unit so that, for example, the worker can easily and surely recognize the safety area. Can be displayed on the projection surface (for example, the floor surface FLR). That is, in this display control system, learning processing and prediction processing are performed using an image taken from above where the shielding area is small, so that the robot arm is in any state according to the state (or). The prediction process for dynamically specifying (predicting) the safety area can be performed appropriately and with high accuracy (according to the transition status of the operation phase of the robot arm specified from the control sequence).

Therefore, in this display control system, the safety area can be appropriately displayed when the robot arm and a movable object (for example, a human being) collaborate. As a result, it is possible to improve work safety while ensuring high work efficiency.

The "movable object" is a movable object, for example, a human or an animal that can move spontaneously.

The second invention is the first invention, in which the prediction processing unit includes a plurality of image regions having colors specified according to the degree of safety based on control data for controlling the robot arm. When the training projection image data generated as described above is projected onto the projection surface, the prediction process is executed using the trained model obtained by executing the training process using the image captured by the imaging unit. To do.

As a result, in this display control system, it is possible to execute the learning process using the images hierarchically color-coded according to the degree of safety as training data, and to execute the prediction process using the acquired learned model.

The third invention is the first invention, in which the prediction processing unit includes a plurality of image regions having brightness specified according to the degree of safety based on control data for controlling the robot arm. When the training projection image data generated as described above is projected onto the projection surface, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit. To do.

As a result, in this display control system, the learning process is executed using the image whose brightness is changed hierarchically according to the degree of safety as training data, and the prediction process can be executed using the acquired model.

The fourth invention is any one of the first to third inventions, in which the prediction processing unit generates a plurality of image regions having colors specified according to the degree of safety and forms them from projected image data. The projected image data is generated so that the image to be generated includes the generated plurality of image regions.

As a result, in this display control system, the image (projected image) indicating the safety area can be made into an image consisting of a plurality of image areas hierarchically color-coded according to the degree of safety, and the worker is projected on the projection surface. The degree of safety can be appropriately recognized by the color.

The color of the plurality of image areas may be one that changes stepwise according to the degree of safety (an image in which the gradation value of each pixel takes a discrete value), or one that changes continuously. (An image in which the gradation value of each pixel takes a continuous value (for example, a gradation image)) may be used.

The fifth invention is any one of the first to third inventions, in which the projection unit generates a plurality of image regions having brightness specified according to the degree of safety, and is formed by the projected image data. The projected image data is generated so that the image includes a plurality of generated image areas.

As a result, in this display control system, an image (projected image) indicating a safety area can be made into an image consisting of a plurality of image areas hierarchically divided by brightness (brightness) according to the degree of safety. The degree of safety can be appropriately recognized by the brightness (brightness) projected on the projection surface.

The brightness (brightness) of the plurality of image areas may change stepwise according to the degree of safety (an image in which the gradation value of each pixel takes a discrete value), or continuously. It may be an image that changes (an image in which the gradation value of each pixel takes a continuous value (for example, an image in which the brightness (brightness) continuously changes)).

In the sixth invention, in a space where a robot arm and a movable object may coexist, the movable object can move in a dangerous area, which is an area determined to be dangerous if the movable object exists. It is a display control system for displaying on a recognizable projection surface, and includes an imaging unit, a prediction processing unit, and a projection unit.

The imaging unit is installed at a position above the robot arm.

The prediction processing unit is (1) an image captured from a position above the robot arm by the imaging unit in space, and is an image captured when the robot arm is in a predetermined state, or the robot arm is predetermined. The learning process is executed using data including control data for controlling the robot arm so as to be in the state of (2) information for identifying a dangerous area when the robot arm is in the predetermined state as teacher data. Prediction processing is executed using the trained model acquired in the above. Further, the prediction processing unit executes prediction processing using the trained model on the prediction processing image which is an image captured from a position above the robot arm by the imaging unit at the time of prediction processing. , The danger area in the space when the image for prediction processing is acquired is predicted, the predicted danger area is acquired as the prediction danger area, and the projected image data is generated based on the prediction danger area.

In this display control system, an image (for example, a frame image) captured by an imaging unit installed above the robot arm, for example, the position and state of the robot arm, the position of a movable object (for example, a worker), and the like. An image that can determine the state (for example, a frame image) or control data for controlling the robot arm so that the robot arm is in a predetermined state, and (2) when the robot arm is in the predetermined state. Prediction processing is performed using the trained trained model using the data that identifies the dangerous area of the robot as training data (teacher data). Then, in this display control system, when the input image is captured by performing the prediction processing by the trained model using the image (for example, the frame image) captured in the same state as when learning. It is possible to predict (identify) a dangerous area (dangerous area on the projection surface). Then, in this display control system, the predicted (identified) dangerous area is projected onto the projection surface (for example, the floor surface) by the projection unit so that, for example, the worker can easily and surely recognize the dangerous area. Can be displayed on the projection surface (for example, the floor surface FLR). That is, in this display control system, since the learning process and the prediction process are performed using the image taken from above where the shielding area is small, the robot arm Rbt_arm is in any state according to the state (or , According to the transition status of the operation phase of the robot arm identified from the control sequence), the prediction process for dynamically identifying (predicting) the dangerous area can be performed appropriately and with high accuracy.

Therefore, in this display control system, when the robot arm and a movable object (for example, a human being) collaborate, the dangerous area can be appropriately displayed. As a result, it is possible to improve work safety while ensuring high work efficiency.

The "movable object" is a movable object, such as a human being or an animal.

A seventh aspect of the invention is the sixth aspect, wherein the prediction processing unit includes a plurality of image regions having colors specified according to the degree of danger based on control data for controlling the robot arm. When the training projection image data generated as described above is projected onto the projection surface, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit. To do.

As a result, in this display control system, it is possible to execute the learning process using the images hierarchically color-coded according to the degree of danger as training data, and to execute the prediction process using the acquired learned model.

The eighth invention is the sixth invention, in which the prediction processing unit includes a plurality of image regions having brightness specified according to the degree of danger based on control data for controlling the robot arm. When the training projection image data generated as described above is projected onto the projection surface, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit. To do.

As a result, in this display control system, the learning process is executed using the image whose brightness is changed hierarchically according to the degree of danger as training data, and the prediction process can be executed using the acquired model.

The ninth invention is any one of the sixth to eighth inventions, in which the prediction processing unit generates a plurality of image regions each having a color specified according to the degree of danger and forms them from the projected image data. The projected image data is generated so that the image to be generated includes a plurality of generated image areas.

As a result, in this display control system, the image (projected image) indicating the dangerous area can be made into an image consisting of a plurality of image areas hierarchically color-coded according to the degree of danger, and the worker is projected on the projection surface. The degree of danger can be appropriately recognized by the color.

The colors of the plurality of image areas may be those that change stepwise according to the degree of danger (images in which the gradation values of each pixel take discrete values), and those that change continuously. (An image in which the gradation value of each pixel takes a continuous value (for example, a gradation image)) may be used.

A tenth invention is any one of the sixth to eighth inventions, in which the prediction processing unit generates a plurality of image regions having brightness specified according to the degree of danger and forms them from projected image data. The projected image data is generated so that the image to be generated includes a plurality of generated image regions.

As a result, in this display control system, the image (projected image) indicating the dangerous area can be made into an image consisting of a plurality of image areas hierarchically divided by the brightness (brightness) according to the degree of danger. The degree of danger can be appropriately recognized by the brightness (brightness) projected on the projection surface.

The brightness (brightness) of the plurality of image regions may change stepwise according to the degree of danger (an image in which the gradation value of each pixel takes a discrete value), or continuously. It may be an image that changes (an image in which the gradation value of each pixel takes a continuous value (for example, an image in which the brightness (brightness) continuously changes)).

The eleventh invention is any one of the first to tenth inventions, in which the space has a floor surface and the projection surface is the floor surface in the space.

As a result, in this display control system, the projection surface can be the floor surface.

The twelfth invention is any one of the first to eleventh inventions, in which the space has a ceiling surface, and the imaging unit is installed on the ceiling surface of the space.

As a result, in this display control system, an imaging unit installed on the ceiling surface can be used.

The thirteenth invention is any one of the first to twelfth inventions, further including a robot arm control unit for controlling the robot arm. Then, when the prediction processing unit determines that the movable object is likely to move out of the safe area or the movable object is likely to move into the dangerous area, the robot arm control unit moves the robot arm. Executes a process that stops the operation and / or generates a warning.

As a result, in this display control system, when a movable object moves from the inside of the safety area to the outside of the safety area and is likely to come into contact with or collide with the robot arm, it comes into contact with the robot arm by a process of avoiding danger. , Collision and other serious accidents can be prevented. That is, in this display control system, regardless of the state of the robot arm, it is possible to appropriately and accurately perform the prediction process of dynamically identifying (predicting) the safety area according to the state. In addition, appropriate risk avoidance processing can be performed.

The fourteenth invention includes an imaging unit installed at a position above the robot arm in a space where a robot arm and a movable object may coexist, a projection unit that projects an image onto a predetermined projection surface, and the like. This is a display control method used in a display control system including. The display control method is a method for displaying a safety area, which is an area determined to be safe even if a movable object exists, on a projection surface in which the movable object can be recognized in space. It includes a processing step and a projection step.

In the prediction processing step, in (1) space, an image captured from a position above the robot arm by the imaging unit, the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing is performed using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for specifying a safety area when the robot arm is in the predetermined state as teacher data. Prediction processing is executed using the trained model acquired by execution. Then, in the prediction processing step, at the time of prediction processing, the imaging unit executes prediction processing using the trained model on the image for prediction processing which is an image captured from a position above the robot arm. , The safety area in the space when the image for prediction processing is acquired is predicted, the predicted safety area is acquired as the prediction safety area, and the projected image data is generated based on the prediction safety area.

In the projection step, the projected image formed by the projected image data is projected onto the projection surface.

Thereby, a display control method having the same effect as that of the first invention can be realized.

A fifteenth invention includes an imaging unit installed at a position above the robot arm in a space where a robot arm and a movable object may coexist, a projection unit that projects an image onto a predetermined projection surface, and the like. This is a display control method used in a display control system including. The display control method is a method for displaying a dangerous area, which is an area determined to be dangerous when a movable object is present, on a projection surface in which the movable object can be recognized in space. It includes a prediction processing step and a projection step.

In the prediction processing step, in (1) space, an image captured from a position above the robot arm by the imaging unit, the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing is performed using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for identifying a dangerous area when the robot arm is in the predetermined state as teacher data. Prediction processing is executed using the trained model acquired by execution. Then, in the prediction processing step, at the time of prediction processing, the image pickup unit executes prediction processing using the trained model on the prediction processing image which is an image captured from a position above the robot arm. , The danger area in the space when the image for prediction processing is acquired is predicted, the predicted danger area is acquired as the prediction danger area, and the projected image data is generated based on the prediction danger area.

Thereby, a display control method having the same effect as that of the sixth invention can be realized.

According to the present invention, a technique for improving work safety while ensuring high work efficiency by appropriately displaying a safety area (or a danger area) when a robot arm and a human perform joint work. Can be realized.

The schematic block diagram of the display control system 1000 which concerns on 1st Embodiment. The schematic block diagram of the display control device 100 which concerns on 1st Embodiment. The figure for demonstrating the processing of the training data acquisition mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the training data acquisition mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the learning mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 1000 which concerns on 1st Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 1000 which concerns on 1st Embodiment. The schematic block diagram of the display control system 1000A which concerns on 1st modification of 1st Embodiment. The schematic block diagram of the display control device 100A which concerns on 1st modification of 1st Embodiment. The figure for demonstrating the processing of the training data acquisition mode of the display control system 1000A which concerns on 1st modification of 1st Embodiment. The figure (timing chart) which shows an example (pattern 1) of the training data used in the display control system 1000A which concerns on the 1st modification of 1st Embodiment. FIG. 6 is a diagram (timing chart) showing an example (pattern 2) of training data used in the display control system 1000A according to the first modification of the first embodiment. The schematic block diagram of the display control system 2000 which concerns on 2nd Embodiment. The schematic block diagram of the display control device 100A which concerns on 2nd Embodiment. The schematic block diagram of the prediction processing part 5A of the display control apparatus 100A which concerns on 2nd Embodiment. It is a flowchart of the process of the prediction mode of the display control system 2000 which concerns on 2nd Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 2000 which concerns on 2nd Embodiment. The figure for demonstrating the processing of the prediction mode of the display control system 2000 which concerns on 2nd Embodiment. The figure which shows the CPU bus configuration.

[First Embodiment]
The first embodiment will be described below with reference to the drawings.

<1.1: Display control system configuration>
FIG. 1 is a schematic configuration diagram of the display control system 1000 according to the first embodiment.

FIG. 2 is a schematic configuration diagram of the display control device 100 according to the first embodiment.

As shown in FIG. 1, the display control system 1000 includes a projection unit Prj1, an imaging unit Cmr1, a display control device 100, and a robot Rbt.

The projection unit Prj1 is installed at a position higher than the highest point (for example, the ceiling) of the worker or the robot Rbt, and is an image or a boundary line or the like downward from the projection point P_prj (for example, with respect to the floor surface FLR). It is a device that projects. The projection unit Prj1 is realized by, for example, a projector device that projects an image onto the floor surface FLR, or a device that can display a line of a predetermined color on the floor surface such as an LED scanner or a laser scanner.

The projection unit Prj1 inputs the control signal Ctl_prj (D_prj) output from the display control device 100, and based on the control signal Ctl_prj (D_prj), the projection target (for example, the floor surface) has an image or a boundary line. Etc. (data D_prj) are projected.

The image pickup unit Cmr1 is installed at a position higher than the highest point of the worker or the robot Rbt (for example, the ceiling), captures an image projected by the projection unit Prj1 on the floor surface, a display of a boundary line, or the like, and also a worker ( For example, the worker Psn1 of FIG. 1, the helmet worn by the worker (for example, the helmet Hat1 worn by the worker Psn1 of FIG. 1), or the marker worn by the worker (for example, Infrared light reflection marker) etc. are imaged. The imaging unit Cmr1 is equipped with, for example, an image sensor for visible light and is equipped with an imaging device capable of capturing a color image, an image sensor for infrared light, and an infrared camera capable of capturing an infrared light image. It will be realized.

The imaging unit Cmr1 outputs the captured data (image data, video data) as data D1_img to the display control device 100.

As shown in FIG. 2, the display control device 100 includes a selector SEL1, a projection control unit 1, a selector SEL2, a training data acquisition unit 2, a training data storage unit DB1, and a CG synthesis unit 3 (CG: Computer Graphics). ), The extended training data storage unit DB2, the learning unit 4, the optimization parameter storage unit DB3, and the prediction processing unit 5.

The selector SEL1 is output from the data D1_prj_train including the projection data at the time of training data acquisition and the prediction processing unit 5 according to the mode signal model output from the control unit (not shown) that controls each functional unit of the display control device 100. This is a selector that selects one of the data Dp1_prj including the projection data at the time of prediction and outputs the selected data as the data D_prj to the projection control unit 1. Specifically, (1) at the time of training data acquisition, the signal value of the mode signal Mode is set to "0", and the selector SEL1 selects the data D1_prj_train including the projection data at the time of training data acquisition. , The data is output to the projection control unit 1 as data D_prj. (2) At the time of prediction, the signal value of the mode signal Mode is set to "1", and the selector SEL1 selects the data Dp1_prj including the projection data at the time of prediction output from the prediction processing unit 5. The data Dp1_prj is output to the projection control unit 1 as the data D_prj.

The projection control unit 1 inputs the data D_prj output from the selector SEL1, and the projection data (image data for projection or boundary line data to be displayed on the projection surface) included in the data D_prj is the projection unit. A control signal Ctl_prj (D_prj) that controls the projection from Prj1 so as to be projected onto the projection target is generated, and the generated control signal Ctl_prj (D_prj) is output to the projection unit Prj1.

The selector SEL2 obtains the data D1_img output from the imaging unit Cmr1 according to the mode signal model output from the control unit (not shown) that controls each functional unit of the display control device 100, to the training data acquisition unit 2 and the prediction processing unit. This is a selector that outputs to any one of 5. Specifically, (1) at the time of training data acquisition, the signal value of the mode signal Mode is set to "0", and the selector SEL2 outputs the data D1_img to the training data acquisition unit 2. (2) At the time of prediction, the signal value of the mode signal Mode is set to "1", and the selector SEL2 outputs the data D1_img to the prediction processing unit 5.

The training data acquisition unit 2 inputs the data D1_img output from the selector SEL2, and generates training data Dtr1 for training the learning model of the prediction processing unit 5 based on the data D1_img. Then, the training data acquisition unit 2 stores the generated training data Dtr1 in the training data storage unit DB1.

The training data storage unit DB1 stores the training data Dtr1 output from the training data acquisition unit 2 in accordance with the instruction from the training data acquisition unit 2. Further, the training data storage unit DB1 outputs the stored training data Dtr1 to the CG synthesis unit 3 in accordance with the instruction from the CG synthesis unit 3.

The CG synthesis unit 3 reads (acquires) the training data Dtr1 from the training data storage unit DB1. Then, the CG synthesis unit 3 creates training data synthesized by CG using the training data Dtr1, and stores the created data as the extended training data Dtr2 in the extended training data storage unit DB2. The CG synthesis unit 3 may also include the training data Dtr1 used for creating the extended training data Dtr2 in the extended training data Dtr2 and store it in the extended training data storage unit DB2.

The extended training data storage unit DB2 stores the extended training data Dtr2 output from the CG synthesis unit 3 in accordance with the instruction from the CG synthesis unit 3. Further, the extended training data storage unit DB2 outputs the stored extended training data Dtr2 to the learning unit 4 in accordance with the instruction from the learning unit 4.

The learning unit 4 reads (acquires) the extended training data Dtr2 from the extended training data storage unit DB2, and performs learning processing using the extended training data Dtr2. Then, the learning unit 4 acquires a parameter (optimization parameter θ_opt) for optimizing the learning model by the learning process, and stores the acquired optimization parameter θ_opt in the optimization parameter storage unit DB3.

The optimization parameter storage unit DB3 stores the optimization parameter θ_opt output from the learning unit 4 in accordance with the instruction from the learning unit 4. Further, the optimization parameter storage unit DB3 outputs the stored optimization parameter θ_opt to the prediction processing unit 5 in accordance with the instruction from the prediction processing unit 5.

The prediction processing unit 5 reads the optimization parameter θ_opt from the optimization parameter storage unit DB3, and acquires a trained model based on the optimization parameter θ_opt. The prediction processing unit 5 inputs the data D1_img output from the selector SEL2, and executes the prediction processing using the trained model for the data D1_img. Then, the prediction processing unit 5 generates data Dp1_prj (data projected from the projection unit Prj1) to be output to the projection control unit 1 based on the prediction processing result. Then, the prediction processing unit 5 outputs the generated data Dp1_prj to the selector SEL1.

The robot Rbt includes a robot control unit Rbt_C1 and a robot arm Rbt_arm.

The robot control unit Rbt_C1 is a functional unit for controlling the robot arm Rbt_arm. The robot control unit Rbt_C1 inputs training data D1_rb_train and a predetermined control signal, and generates a control signal for controlling the robot arm Rbt_arm based on the input training data D1_rb_train and a predetermined control signal.

The robot arm Rbt_arm performs a predetermined operation (for example, grasping, transporting, etc.) based on a command (control signal) from the robot control unit Rbt_C1.

<1.2: Operation of display control system>
The operation of the display control system 1000 configured as described above will be described below.

In the following, the operation of the display control system 1000 will be described separately for (1) training data acquisition mode processing, (2) learning mode processing, and (3) prediction mode processing.

For convenience of explanation, the display control system 1000 is installed in a narrow space (indoor space) such as a factory, and the projection unit Prj1 and the imaging unit Cmr1 are installed on the ceiling, and the worker Psn1 and the robot Rbt Will be described below assuming that is present on the floor surface. Further, the projection unit Prj1 targets the floor surface FLR as a projection target.

(1.2.1: Training data acquisition mode processing)
First, the processing of the training data acquisition mode will be described.

3 and 4 are diagrams for explaining the processing of the training data acquisition mode of the display control system 1000 according to the first embodiment.

As shown in FIG. 3, the robot control unit Rbt_C1 inputs data D1_rb_train for causing the robot arm Rbt_arm to execute a predetermined operation (this is referred to as a first operation) in order to acquire training data. Further, it is safe even if the projection control unit 1 of the display control device 100 has a safety area when the robot arm Rbt_arm is first operated, that is, a worker (for example, worker Psn1 in FIG. 3) exists. Input the data D1_prj_train for displaying the area (the area where the worker can safely perform the work without touching the robot arm Rbt_arm during the first operation) on the floor surface to be projected. ..

The robot arm Rbt_arm performs the first operation based on the data D1_rb_train. During the period in which the robot arm Rbt_arm is performing the first operation, the projection unit Prj1 is performing the first operation when the robot arm Rbt_arm is performing the first operation based on the data D1_prj_train input to the projection control unit 1 of the display control device 100. An image or a boundary line showing the safety area of the robot is displayed on the floor surface FLR to be projected. In the display control device 100, for example, the projection control unit 1 controls the projection unit Prj1 so as to project a predetermined test image including a test pattern having a known size on the image onto the floor surface. , The distance from the projection point P_prj of the projection unit Prj1 to the projection surface (floor surface FLR) is acquired by examining the size of the test pattern in the image (image captured image) obtained by the imaging unit Cmr1 of the test image. .. It is assumed that the imaging parameters (angle of view, focal length, etc.) of the imaging unit Cmr1 are known. Then, the display control device 100 controls the projection control unit 1 so that the image showing the safety area or the boundary line is projected on the floor surface FLR which is the projection surface based on the acquired distance.

In the case shown in FIG. 3, the area Ar1 is a safety area when the robot arm Rbt_arm is performing the first operation, and a boundary line indicating the safety area is projected from the projection unit Prj1 onto the floor surface FLR.

In this state, the worker Psn1 is asked to wear a helmet Hat1 (for example, a yellow helmet) to work or move in the safe area (in the area Ar1), and the situation at that time is captured by the imaging unit Cmr1. Take an image with. For convenience of explanation, the imaging unit Cmr1 will be described below assuming that the imaging unit Cmr1 is equipped with an image sensor for visible light and is capable of capturing a color image. Further, the imaging unit Cmr1 has camera parameters (optical axis direction, angle of view, focal length) so that the safety region (region Ar1 in the case of FIG. 3) and the robot arm Rbt_arm are included in the captured image (captured image). Etc.) shall be adjusted.

The imaging unit Cmr1 outputs an image of the situation when the robot arm Rbt_arm is performing the first operation as an image (frame image) (data D1_img) for each frame to the training data acquisition unit 2.

At the time of training data acquisition, the signal value of the mode signal Mode is set to "0", so the selector SEL2 outputs the data D1_img to the training data acquisition unit 2.

The training data acquisition unit 2 inputs the data D1_img output from the selector SEL2, and generates training data Dtr1 for training the learning model of the prediction processing unit 5 based on the data D1_img. Specifically, the training data acquisition unit 2 stores the data D1_img (frame image data) as the training data Dtr1 as the actual image data in the training data storage unit DB1. Further, the training data acquisition unit 2 sets, for example, an image obtained by extracting a predetermined image feature amount from the data D1_img (frame image data) (for example, the image feature amount is the same color as the color of the helmet worn by the worker). Even if an image obtained by extracting the image area of the color portion) is acquired and combined with the original image D1_img (frame image data) (actual image data), and the combined image is generated as training data Dtr1. Good.

Further, the training data acquisition unit 2 may include the image data and the information (additional information (label information)) of the state when the image is acquired in the training data Dtr1. For example, information about the operation phase of the robot arm Rbt_arm, information indicating that the safety area is correct in the area Ar1, and the like may be included in the training data Dtr1 together with the image data as additional information.

The training data acquisition unit 2 stores the generated training data Dtr1 in the training data storage unit DB1.

The CG synthesis unit 3 reads (acquires) the training data Dtr1 from the training data storage unit DB1. Then, the CG synthesis unit 3 creates the training data synthesized by CG using the training data Dtr1. For example, in the image data of the training data Dtr1, the CG synthesis unit 3 generates a CG image area in which the image area of the helmet portion of the worker is changed to a color different from the color of the helmet by CG processing, and the CG image area is used. , CG image data is generated by replacing the image area of the helmet portion of the actual image. That is, this CG image data becomes an image in which the color of the helmet portion is changed by the CG processing. In the same manner as above, the CG synthesis unit 3 performs CG processing (CG synthesis) such as changing the helmet color to another color or texture, or replacing the helmet color with the worker's hair (when the helmet is not worn). Processing) generates various CG composite images from the original image. By processing in this way, a large amount of training data can be acquired from one training data Dtr1. Then, the CG synthesis unit 3 adds the CG image data generated as described above to the original image data to generate the extended training data Dtr2. Then, the CG synthesis unit 3 stores the generated extended training data Dtr2 in the extended training data storage unit DB2.

As a method for acquiring a large amount of image data (training image data) from the original image data by CG processing (CG synthesis processing), the method described in Japanese Patent Application No. 2019-008307 may be used.

By processing as described above, the display control system 1000 acquires training data (learning data) for the case of FIG. 3 (when the robot arm Rbt_arm is performing the first operation).

Further, in the display control system 1000, the state when the robot arm Rbt_arm is performing the second operation different from the first operation is imaged, and the training data is further acquired.

As shown in FIG. 4, in order to acquire training data, the robot control unit Rbt_C1 inputs data D2_rb_train for causing the robot arm Rbt_arm to execute a second operation different from the first operation. Further, it is safe even if the projection control unit 1 of the display control device 100 has a safety area when the robot arm Rbt_arm is subjected to the second operation, that is, a worker (for example, worker Psn1 in FIG. 4) exists. Input data D2_prj_train to display the area (the area where the worker can safely work without touching the robot arm Rbt_arm during the second operation) on the floor surface to be projected. ..

The robot arm Rbt_arm performs the second operation based on the data D2_rb_train. During the period when the robot arm Rbt_arm is performing the second operation, the projection unit Prj1 is performing the second operation when the robot arm Rbt_arm is performing the second operation based on the data D2_prj_train input to the projection control unit 1 of the display control device 100. An image or a boundary line showing the safety area of the robot is displayed on the floor surface FLR to be projected. In the case shown in FIG. 4, the area Ar2 is a safety area when the robot arm Rbt_arm is performing the second operation, and the boundary line indicating the safety area is projected from the projection unit Prj1 onto the floor surface FLR.

In this state, the worker Psn1 is asked to wear a helmet Hat1 (for example, a yellow helmet) to work or move in the safe area (in the area Ar1), and the situation at that time is captured by the imaging unit Cmr1. Take an image with.

Then, by executing the same process as the process of acquiring the training data when the robot arm Rbt_arm is performing the first operation, the display control system 1000 performs the process when the robot arm Rbt_arm is performing the second operation. The extended training data Dtr2 is acquired, and the extended training data Dtr2 is stored in the extended training data storage unit DB2.

As described above, in the display control system 1000, the processing of the training data acquisition mode is executed, and the training data (learning data) (extended training data Dtr2) is acquired.

In the above description, in the display control system 1000, in order to acquire training data, an image showing a safety area on the floor surface or a boundary line is projected from the projection unit Prj1 and displayed on the floor surface. There is no limitation. For example, in the display control system 1000, an image (frame image) is acquired by the imaging unit Cmr1 without projecting an image or a boundary line showing the safety area on the floor surface from the projection unit Prj1, and the acquired frame image is the safety area. The image or boundary line indicating the above may be synthesized (for example, by CG processing) to generate training data (image data for training). Further, the safety area may be specified manually or automatically (for example, by calculation) depending on the position and state of the worker and the robot arm Rbt_arm in the acquired frame image.

(1.2.2: Learning mode processing)
Next, the processing of the learning mode will be described.

FIG. 5 is a diagram for explaining the processing of the learning mode of the display control system 1000 according to the first embodiment.

As shown in FIG. 5, the learning unit 4 reads (acquires) the extended training data Dtr2 from the extended training data storage unit DB2, and performs learning processing using the extended training data Dtr2. The learning unit 4 performs learning processing using the extended training data Dtr2 as teacher data. Specifically, the learning unit 4 uses the input as image data included in the extended training data Dtr2 (this image data is referred to as "Dtr2.img"), and outputs the output as a safety area when the image data is acquired ( As information for specifying the safety area to be displayed on the projection surface), learning is performed on a learning model (for example, a model realized by a neural network). The learning model is, for example, a model by a neural network including an input layer, a plurality of intermediate layers, and an output layer. It is assumed that the weighting coefficient (weighting of synaptic connections connecting each layer) between each layer of the training model is set (adjusted) by the parameter θ.

The learning unit 4 has input data to the learning model Dtr2. Let x be the set of img, let y be the set of output data from the training model, and P (y | x) be the conditional probability that the output data y will be output when the input data x is input to the training model. Then

The optimum parameter θ_opt that satisfies the condition is acquired by repeating the process of updating (adjusting) the parameters of the training model. It should be noted that the conditional P (y | x) takes a larger value as the output data is closer to the teacher data.

For example, conditional P (y | x) is set as follows.

σ: Standard deviation Note that x _i is a vector included in the set x (data obtained by converting the data of each pixel of the two-dimensional image into a one-dimensional vector), and y _i is a vector included in the set y, y. _{i_select} is teacher data (correct answer data) (vector data) when _xi is input. H (x _i ; θ) represents an operator corresponding to, for example, processing of a neural network composed of a plurality of layers on the input x _i and acquiring an output. The parameter θ is, for example, a parameter that determines the weighting of synaptic connections of the neural network. Incidentally, _H; the (x i θ), may include the calculation of the nonlinear.

When the learning unit 4 performs learning processing using the learning model using the extended training data and determines that it has sufficiently converged, the parameter θ set in the learning model at that time is set as the optimization parameter θ_opt. It is acquired, and the acquired optimization parameter θ_opt is stored in the optimization parameter storage unit DB3.

As described above, the display control system 1000 executes the learning mode processing.

(12.3: Prediction mode processing)
Next, the processing of the prediction mode will be described.

6 to 9 are diagrams for explaining the processing of the prediction mode of the display control system 1000 according to the first embodiment.

As shown in FIG. 6, a case where a control signal Ctrl_Rbt (phase1) is input to the robot control unit Rbt_C1 and the robot arm Rbt_arm executes a predetermined operation (this is referred to as “phase 1 operation”) will be described.

In this case, the robot control unit Rbt_C1 controls the robot arm Rbt_arm so that the robot arm Rbt_arm executes an operation (phase 1 operation) according to the control signal Ctrl_Rbt (phase1).

The robot arm Rbt_arm performs the operation of Phase 1 in accordance with a command from the robot control unit Rbt_C1.

Then, the imaging unit Cmr1 captures the situation at that time (robot arm Rbt_arm, worker, floor surface FLR, etc.) in the same state as when the training data was acquired. Since the positional relationship between the projection unit Prj1, the imaging unit Cmr1, the robot Rbt, and the floor FLR may deviate, calibration (relative positional relationship adjustment processing) should be performed before executing the prediction mode processing. You may. For example, two different points (for example, points P1 and P2 shown in FIG. 6) on the pedestal of the robot Rbt existing on a plane parallel to the floor surface FLR are detected and detected in the image captured by the imaging unit Cmr1. The direction of the optical axis of the imaging unit Cmr1 and the camera angle so that the image corresponding to the two points corresponds to the position corresponding to the two points of the image when the training data is acquired in the display control device 100. Adjust the angle of view, etc. By this calibration (adjustment process of relative positional relationship), in the display control system 1000, the image acquired by the imaging unit Cmr1 has the same situation as the image when the training data is acquired, that is, the positional relationship in the image. Since the same image is obtained, the accuracy of the prediction process is improved. In addition, in order to improve the accuracy at the time of calibration, a point group of 3 or more points is used as a calibration point, and each calibration parameter (direction of optical axis, camera angle, angle of view, etc.) generated by the point group is used. ) May be used.

After performing the calibration, while the robot arm Rbt_arm is performing the phase 1 operation, the imaging unit Cmr1 images the robot Rbt, the floor surface FLR, and the worker (if any) from above and images the image. The image (frame image) is continuously output to the display control device 100.

At the time of prediction processing (prediction mode), the signal value of the mode signal Mode is set to "1", so the selector SEL2 outputs the data D1_img to the prediction processing unit 5.

When the processing mode of the display control device 100 is set to the prediction mode, the prediction processing unit 5 reads the optimization parameter θ_opt from the optimization parameter storage unit DB3 and acquires a trained model based on the optimization parameter θ_opt. To do. That is, the prediction processing unit 5 sets the parameter θ in the learning model by the optimization parameter θ_opt. As a result, the trained model is constructed, and the prediction processing unit 5 causes the trained model to input the image data D1_img output from the selector SEL2.

The prediction processing unit 5 inputs the image data D1_img acquired during the period during which the robot arm Rbt_arm is performing the operation of Phase 1 into the trained model, and acquires the output from the trained model as data Dp1_prj. For example, in the case shown in FIG. 6, (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 are the training data in the case of FIG. 3 (when the safety area is the area Ar1). It is similar to (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 in the state when the above is acquired. Therefore, in this state, when the captured image data D1_img is input to the trained model of the prediction processing unit 5, data indicating that the safety region is the region Ar1 (for displaying the region Ar1 on the floor surface FLR). Data Dp1_prj) is output from the trained model.

Then, the prediction processing unit 5 outputs the acquired data Dp1_prj to the selector SEL1. In the prediction mode, since the signal value of the mode signal Mode is “1”, the selector SEL1 outputs the data Dp1_prj output from the prediction processing unit 5 to the projection control unit 1.

The projection control unit 1 inputs data Dp1_prj and outputs a control signal Ctrl_prj (Dp1_prj) that controls the image based on the data Dp1_prj (or the boundary line displayed on the projection surface) to be projected from the projection unit Prj1 onto the projection target. The generated control signal Ctrl_prj (Dp1_prj) is output to the projection unit Prj1.

Then, the projection unit Prj1 projects an image (or a boundary line to be displayed on the projection surface) based on the data Dp1_prj on the projection surface (floor surface FLR) based on the control signal Ctl_prj (Dp1_prj). When shown in FIG. 6, an image (or boundary line) indicating that the safety region is the region Ar1 is projected on the floor surface FLR.

The worker Psn1 can easily determine where the safety area (area Ar1 in the case of FIG. 6) is from the image (or boundary line) projected on the floor surface FLR, and works in the safety area. It does not interfere with the operation of the robot arm Rbt_arm, or does not come into contact with or collide with the robot arm Rbt_arm by moving within the safe area. Therefore, safety is ensured. Further, since the operation of the robot arm Rbt_arm does not come into contact with or collide with the worker, the high-speed operation can be continued, and as a result, the work efficiency by the robot arm Rbt_arm can be maintained in a high state.

Next, as shown in FIG. 7, a case where a control signal Ctrl_Rbt (phase2) is input to the robot control unit Rbt_C1 and the robot arm Rbt_arm executes a predetermined operation (this is referred to as “phase 2 operation”) explain.

In this case, the robot control unit Rbt_C1 controls the robot arm Rbt_arm so that the robot arm Rbt_arm executes an operation (phase 2 operation) according to the control signal Ctrl_Rbt (phase2).

The robot arm Rbt_arm performs the operation of Phase 2 in accordance with a command from the robot control unit Rbt_C1.

Then, the imaging unit Cmr1 captures the situation at that time (robot arm Rbt_arm, worker, floor surface FLR, etc.) in the same state as when the training data was acquired.

During the period in which the robot arm Rbt_arm is performing the phase 2 operation, the imaging unit Cmr1 images the robot Rbt, the floor surface FLR, and the worker (if any) from above, and displays the captured image (frame image). Continue to output to the control device 100.

The prediction processing unit 5 inputs the image data D1_img acquired during the period during which the robot arm Rbt_arm is performing the operation of Phase 2 into the trained model, and acquires the output from the trained model as data Dp1_prj. For example, in the case shown in FIG. 7, (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 are the training data in the case of FIG. 4 (when the safety area is the area Ar2). It is similar to (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 in the state when the above is acquired. Therefore, in this state, when the captured image data D1_img is input to the trained model of the prediction processing unit 5, data indicating that the safety region is the region Ar2 (for displaying the region Ar2 on the floor surface FLR). Data Dp1_prj) is output from the trained model.

The projection control unit 1 inputs the data Dp1_prj and outputs a control signal Ctrl_prj (Dp1_prj) that controls the image (or the boundary line displayed on the projection surface) based on the data Dp1_prj to be projected from the projection unit Prj1 onto the projection target. The generated control signal Ctrl_prj (Dp1_prj) is output to the projection unit Prj1.

Then, the projection unit Prj1 projects an image (or a boundary line to be displayed on the projection surface) based on the data Dp1_prj on the projection surface (floor surface FLR) based on the control signal Ctl_prj (Dp1_prj). When shown in FIG. 7, an image (or boundary line) indicating that the safety region is the region Ar2 is projected on the floor surface FLR.

The worker Psn1 can easily determine where the safety area (area Ar2 in the case of FIG. 7) is from the image (or boundary line) projected on the floor surface FLR, and works in the safety area. It does not interfere with the operation of the robot arm Rbt_arm, or does not come into contact with or collide with the robot arm Rbt_arm by moving within the safe area. Therefore, safety is ensured. Further, since the operation of the robot arm Rbt_arm does not come into contact with or collide with the worker, the high-speed operation can be continued, and as a result, the work efficiency by the robot arm Rbt_arm can be maintained in a high state.

As described above, in the display control system 1000, the image (frame image) captured by the robot arm Rbt_arm and the imaging unit Cmr1 installed above the worker, and the position, state, and worker of the robot Rbt and the robot arm Rbt_arm. Prediction processing using a trained model is performed using an image (frame image) that can determine the position and state of the robot and data that identifies a safe area when the image is acquired as training data (teacher data). Do. Then, in the display control system 1000, the safety area when the input image is captured is performed by performing the prediction processing by the trained model using the image (frame image) captured in the same state as when learning. (Safety area on the projection plane) can be predicted (specified). Then, in the display control system 1000, the predicted (specified) safety area is projected onto the projection surface (floor surface FLR) by the projection unit Prj1, so that the safety area can be easily and surely recognized by the operator. It can be displayed on a surface (floor surface FLR). That is, in the display control system 1000, since the learning process and the prediction process are performed using the image taken from above where the shielding area is small, the robot Rbt and the robot arm Rbt_arm are in any state according to the state. Therefore, the prediction process for dynamically identifying (predicting) the safety area can be performed appropriately and with high accuracy.

In the above description, the case where the boundary line is displayed on the projection surface (floor surface FLR) has been described, but the present invention is not limited to this, and in the display control system 1000, as shown in FIG. An image having a hierarchically divided image area (for example, a hierarchically color-coded image or a hierarchically changed brightness) may be projected onto a projection surface (floor surface FLR). By doing so, the worker can grasp the degree of safety and recognize the safety area.

Further, in the above, the case where the safety area is displayed from the projection unit Prj1 has been described, but the present invention is not limited to this, and in the display control system 1000, as shown in FIG. 9, an image (or an image showing the danger area) showing the danger area is described. The boundary line) may be displayed on the projection surface (floor surface FLR). In the case of FIG. 9, the area Ar_rb1 is a high-risk area (dangerous area), and the area Ar_rb2 is a lower risk area (dangerous area) than the area Ar_rb1. In this case, the safety of the worker is ensured by performing work, movement, etc. outside the dangerous area.

As described above, in the display control system 1000, the safety area (or the danger area) can be appropriately displayed when the robot arm and the human perform joint work. As a result, it is possible to improve work safety while ensuring high work efficiency.

≪First modification≫
Next, a first modification of the first embodiment will be described.

The same parts as those in the above embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

FIG. 10 is a schematic configuration diagram of the display control system 1000A according to the first modification of the first embodiment.

FIG. 11 is a schematic configuration diagram of the display control device 100A according to the first modification of the first embodiment.

FIG. 12 is a diagram for explaining the processing of the training data acquisition mode of the display control system 1000A according to the first modification of the first embodiment.

FIG. 13 is a diagram (timing chart) showing an example (pattern 1) of training data used in the display control system 1000A according to the first modification of the first embodiment.

FIG. 14 is a diagram (timing chart) showing an example (pattern 2) of training data used in the display control system 1000A according to the first modification of the first embodiment.

As shown in FIG. 10, the display control system 1000A according to the present modification has a configuration in which the display control device 100 is replaced with the display control device 100A in the display control system 1000 of the first embodiment. Then, as shown in FIG. 11, the display control device 100A has a configuration in which the training data acquisition unit 2 is replaced with the training data acquisition unit 2A in the display control device 100 of the first embodiment.

The training data acquisition unit 2A inputs the data D1_img output from the selector SEL2 and the training data D1_rb_train input to the robot control unit Rbt_C1. The training data acquisition unit 2A generates training data Dtr1 for training the learning model of the prediction processing unit 5 based on the data D1_img and the training data D1_rb_train.

In the display control system 1000A of this modification, training data is acquired by associating a predetermined control sequence of the robot Rbt with a safety area (or danger area) determined accordingly. For example, as shown in FIG. 13, the robot arm Rbt_arm has (1) Phase 1 (risk: low), (2) Phase 2 (risk: high), and (3) Phase 3 (risk: low (danger)). If it is predetermined that the degree is controlled to operate in the order of (the degree is the same as the degree of danger in Phase 1)), the safety area (or danger area) is also determined according to the above phase. can do.

For example, in the case of FIG. 13, in the display control system 1000A, the training data D1_rb_train for the robot Rbt is determined from a predetermined control sequence of the robot Rbt, and the degree of risk is determined according to the phase determined by the control sequence. Along with the determination, the safety area (or danger area) is determined. In the case of FIG. 13, since the risk of Phase 2 is higher than the risk of Phase 1, the image corresponding to Phase 2 (the image clearly indicating the safety area or the danger area) is longer than the projection surface (for example, the floor surface FLR). The training data D1_prj_train is generated so that it is projected for a period of time. That is, in the case of FIG. 13, from the time t01 before the start time t1 of the phase 2, the image corresponding to the phase 2 (an image clearly indicating the safety area or the danger area) is projected onto the projection surface (for example, the floor surface FLR). The training data D1_prj_train is generated so as to be performed. Then, accordingly, the image corresponding to Phase 1 (the image clearly indicating the safety area or the danger area) is projected on the projection surface (for example, the floor surface FLR) in the period from time t0 to time t01. The training data D1_prj_train is generated. That is, the training data D1_prj_train is generated so that the projected image is switched from the phase 1 image to the phase 2 image at a time before the time t1 when the phase 1 shifts to the phase 2. By doing so, the worker can recognize that the safety area becomes smaller before shifting to the phase where the safety area becomes smaller, and as a result, the safety of the worker is ensured.

Further, as shown in FIG. 14, training data D1_prj_train so that an image composed of an image region having a color (or brightness) determined according to the degree of risk is projected on a projection surface (for example, a floor surface FLR). May be generated.

For example, in the case of FIG. 14, as in the case of FIG. 13, in the display control system 1000A, the training data D1_rb_train for the robot Rbt is determined from a predetermined control sequence of the robot Rbt, and is determined by the control sequence. The degree of danger is determined according to the phase, and the safety area (or danger area) is determined. In the case of FIG. 14, since the risk of Phase 2 is higher than the risk of Phase 1, the image area corresponding to Phase 1 and the image area corresponding to Phase 2 are used in the period before the transition from Phase 1 to Phase 2. The training data D1_prj_train is generated so that the image (an image in which the safety area or the danger area is hierarchically divided by color or brightness according to the degree of danger) is projected on the projection surface (for example, the floor surface FLR). That is, in the case of FIG. 14, in the period from the time t01 to the time t1 before the start time t1 of the phase 2, the image consisting of the image area corresponding to the phase 1 and the image area corresponding to the phase 2 (safety area or danger). The training data D1_prj_train is generated so that the area (an image in which the area is hierarchically divided by color or brightness according to the degree of danger) is projected on the projection surface (for example, the floor surface FLR). Then, accordingly, the image corresponding to Phase 1 (the image clearly indicating the safety area or the danger area) is projected on the projection surface (for example, the floor surface FLR) in the period from time t0 to time t01. The training data D1_prj_train is generated. That is, from the time t01 before the time t1 when the transition from the phase 1 to the phase 2 occurs, the image consisting of the image area corresponding to the phase 1 and the image area corresponding to the phase 2 (safety area or dangerous area according to the degree of danger). By projecting (images hierarchically divided by color or brightness) onto the projection surface, the operator can properly grasp that the degree of danger will soon change and the safety area will change. Therefore, by learning from the training data generated in this way, it is possible to construct a trained model for prediction processing that appropriately ensures the safety of workers.

In the display control system 1000A of this modification, the projection image associated with the control sequence of the robot Rbt is acquired as training data as described above. Then, the training process is performed using the training data acquired in this way, and the trained model is acquired. Then, by executing the prediction process using the learning model, the display control system 1000A appropriately projects an image clearly indicating the safety area (or danger area) on the projection surface (for example, the floor surface FLR). be able to. As a result, the worker can properly grasp that the safety area will change soon, and the safety of the worker is ensured.

In the display control system 1000A of this modification, for example, training data is generated from an image projected on a projection surface (color and brightness of the image), and learning processing is performed based on the training data. You may. That is, in the display control system 1000A of this modified example, training data is acquired and learned based on the image (color and brightness of the image) projected on the projection surface without recognizing the state of the robot arm Rbt_arm. The process may be performed.

[Second Embodiment]
Next, the second embodiment will be described.

<2.1: Display control system configuration>
FIG. 15 is a schematic configuration diagram of the display control system 2000 according to the second embodiment.

FIG. 16 is a schematic configuration diagram of the display control device 100B according to the second embodiment.

FIG. 17 is a schematic configuration diagram of the prediction processing unit 5A of the display control device 100B according to the second embodiment.

As shown in FIG. 15, the display control system 2000 according to the second embodiment has a configuration in which the display control device 100 is replaced with the display control device 100B in the display control system 1000 of the first embodiment. Then, as shown in FIG. 16, the display control device 100B has a configuration in which the prediction processing unit 5 is replaced with the prediction processing unit 5A in the display control device 100 of the first embodiment.

As shown in FIG. 17, the prediction processing unit 5A includes a prediction unit 51, a detection target position determination unit 52, a safety range map generation unit 53, and a danger determination unit 54. The prediction processing unit 5A outputs the data Dp1_prj of the prediction processing result to the selector SEL1 and the safety range map generation unit 53.

The prediction unit 51 is a functional unit that executes the same processing as the prediction processing unit 5 of the first embodiment.

The detection target position determination unit 52 inputs the image data D1_img output from the selector SEL2, performs image recognition processing on the image data D1_img, and performs an image recognition process on the image data D1_img to obtain an image of an image area corresponding to the detection target (for example, a worker). Identify the top position. Then, the detection target position determination unit 52 outputs the data including the acquired position information on the image of the detection target to the danger determination unit 54 as data D_pos.

The safety range map generation unit 53 inputs the data Dp1_prj output from the prediction unit 51, and from the data Dp1_prj, map information for specifying the safety area (information for specifying the position, size, shape, etc. of the safety area). ) To get. Then, the safety range map generation unit 53 outputs the data including the acquired map information as data D_map to the danger determination unit 54.

The danger determination unit 54 inputs the data D_pos output from the detection target position determination unit 52 and the data D_map output from the safety range map generation unit 53. Based on the data D_pos and the data D_map, the danger determination unit 54 determines whether or not the detection target (for example, a worker) is in the danger area (or is likely to be within a short period of time from the current time). Is determined. Then, when the danger determination unit 54 determines that the detection target (for example, a worker) is in the danger area (or is likely to be within a short period of time from the current time), the danger determination unit 54 sends a warning signal Sigma_wrn. Generated and output the generated warning signal Sigma_wrn to the robot control unit Rbt_C1A.

The robot control unit Rbt_C1A has the same function as the robot control unit Rbt_C1 of the first embodiment, and further inputs a warning signal Sig_wrn output from the display control device 100B. Then, the robot control unit Rbt_C1A determines that it is dangerous to continue the operation of the robot arm Rbt_arm when the warning signal Sig_wrn is input from the display control device 100B, and determines that it is dangerous to continue the operation of the robot arm Rbt_arm. Process) and / or perform risk avoidance processing such as stopping the robot arm Rbt_arm.

<2.2: Operation of display control system>
The operation of the display control system 2000 configured as described above will be described. The description of the same parts as those in the first embodiment will be omitted.

FIG. 18 is a flowchart of processing in the prediction mode of the display control system 2000 according to the second embodiment.

19 and 20 are diagrams for explaining the processing of the prediction mode of the display control system 2000 according to the second embodiment.

Hereinafter, the processing of the prediction mode of the display control system 2000 will be described with reference to the flowchart of FIG. In the display control system 2000, the processing of the training data acquisition mode and the processing of the learning mode are the same as those of the display control system 1000 of the first embodiment.

As shown in FIG. 19, a case where a control signal Ctrl_Rbt (phase2) is input to the robot control unit Rbt_C1A and the robot arm Rbt_arm executes a predetermined operation (this is referred to as “phase 2 operation”) will be described.

In this case, the robot control unit Rbt_C1A controls the robot arm Rbt_arm so that the robot arm Rbt_arm executes an operation (phase 2 operation) according to the control signal Ctrl_Rbt (phase2).

The robot arm Rbt_arm performs the operation of Phase 2 in accordance with a command from the robot control unit Rbt_C1A.

During the period in which the robot arm Rbt_arm is performing the phase 2 operation, the imaging unit Cmr1 images the robot Rbt, the floor surface FLR, and the worker (if any) from above, and displays the captured image (frame image). Continue to output to the control device 100 (step S1).

At the time of prediction processing (prediction mode), the signal value of the mode signal Mode is set to "1", so the selector SEL2 outputs the data D1_img to the prediction processing unit 5A.

The prediction processing unit 5A inputs the image data D1_img acquired during the period during which the robot arm Rbt_arm is performing the operation of Phase 2 into the trained model, and acquires the output from the trained model as data Dp1_prj. For example, in the case shown in FIG. 19, (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 are the training data in the case of FIG. 4 (when the safety area is the area Ar2). It is similar to (1) the position and state of the robot arm Rbt_arm, and (2) the position and state of the worker Psn1 in the state when the above is acquired. Therefore, in this state, when the captured image data D1_img is input to the trained model of the prediction processing unit 5A, the data indicating that the safety region is the region Ar2 (for displaying the region Ar2 on the floor surface FLR). Data Dp1_prj) is output from the trained model.

Then, the prediction processing unit 5A outputs the acquired data Dp1_prj to the selector SEL1. In the prediction mode, since the signal value of the mode signal Mode is “1”, the selector SEL1 outputs the data Dp1_prj output from the prediction processing unit 5A to the projection control unit 1.

Then, the projection unit Prj1 projects an image (or a boundary line to be displayed on the projection surface) based on the data Dp1_prj on the projection surface (floor surface FLR) based on the control signal Ctrl_prj (Dp1_prj). When shown in FIG. 19, an image (or boundary line) indicating that the safety region is the region Ar2 is projected on the floor surface FLR.

In this state, the frame image data D1_img acquired by the imaging unit Cmr1 is input to the detection target position determination unit 52.

The detection target position determination unit 52 performs image recognition processing on the frame image data D1_img to specify the position on the image of the image region corresponding to the detection target (for example, the worker Psn1 in FIG. 19) (step S2). .. Then, the detection target position determination unit 52 outputs the data including the acquired position information on the image of the detection target to the danger determination unit 54 as data D_pos.

The safety range map generation unit 53 inputs the data Dp1_prj output from the prediction unit 51, and from the data Dp1_prj, map information for specifying the safety area (information for specifying the position, size, shape, etc. of the safety area). ) Is acquired (step S3). In the case of FIG. 19, the safety range map generation unit 53 acquires the map information that identifies the safety area Ar2.

Then, the safety range map generation unit 53 outputs the data including the acquired map information as data D_map to the danger determination unit 54.

The danger determination unit 54 determines the detection target (for example, the worker Psn1 in FIG. 19) based on the data D_pos output from the detection target position determination unit 52 and the data D_map output from the safety range map generation unit 53. Determine if you are in the danger zone (or likely to be in a short period of time from the current time). For example, a motion vector to be detected is acquired, and from the motion vector, it is determined whether or not there is a high possibility that the motion vector is within a short period of time from the current time (map collation process, steps S4 and S5).

Then, when the danger determination unit 54 determines that the detection target (for example, a worker) is in the danger area (or is likely to be within a short period of time from the current time), the danger determination unit 54 sends a warning signal Sigma_wrn. Generated and output the generated warning signal Sigma_wrn to the robot control unit Rbt_C1A.

When the warning signal Sig_wrn is input from the display control device 100B, the robot control unit Rbt_C1A determines that it is dangerous to continue the operation of the robot arm Rbt_arm, and performs a warning operation (for example, a process of generating a warning sound). ) And / or, risk avoidance processing such as stopping the robot arm Rbt_arm is performed (step S6).

In the display control system 2000, by processing in this way, for example, in the case of FIG. 19, the worker Psn1 may move from the inside of the safety area Ar2 to the outside of the safety area Ar2 and come into contact with or collide with the robot arm Rbt_arm. When the value becomes high, the danger avoidance process can prevent the occurrence of serious accidents such as contact and collision with the robot arm Rbt_arm. That is, even in the display control system 2000, regardless of the state of the robot Rbt and the robot arm Rbt_arm, the prediction process for dynamically identifying (predicting) the safety area according to the state is appropriately and highly accurate. Can be done.

In the above description, the case where the boundary line is displayed on the projection surface (floor surface FLR) has been described, but the present invention is not limited to this, and in the display control system 2000, the degree of safety is hierarchically divided into image areas ( For example, an image having a hierarchically color-coded image or an image having a hierarchically changed brightness may be projected onto a projection surface (floor surface FLR). In this case, the above-mentioned process of specifying the position of the detection target (step S2) and the process of acquiring the safety range map (step S3) may be performed using the image data including the image area in which the degree of safety is hierarchically divided.

Further, in the above, the case where the safety area is displayed from the projection unit Prj1 has been described, but the present invention is not limited to this, and in the display control system 2000, as shown in FIG. 20, an image (or an image showing the danger area) showing the danger area is described. The boundary line) may be displayed on the projection surface (floor surface FLR). In the case of FIG. 20, the region Ar_rb1 is a region with a high degree of risk (dangerous region), and the region Ar_rb2 is a region with a lower degree of danger than the region Ar_rb1 (dangerous region). In this case, when the detection target (for example, the worker Psn1) enters the danger range or is likely to enter the danger range, the display control device 100B outputs a warning signal Sig_wrn to the robot control unit Rbt_C1A to perform the danger avoidance process. You just have to do.

[Other Embodiments]
The display control system and the display control device may be configured by combining the above embodiments. For example, in the display control system of the second embodiment, training data is acquired by the same method as in the first modification of the first embodiment, learning processing is performed using the acquired training data, and further. The prediction process may be performed using the trained model acquired by the training process.

Further, in the above embodiment, the case where there is one robot Rbt and one worker has been described, but the present invention is not limited to this, and the number of robot Rbts and the number of workers may be any number. ..

Further, in the above embodiment, the case where the imaging unit Cmr1 is one has been described, but the present invention is not limited to this, and the display control system may include a plurality of imaging units. Then, in the display control system configured as described above, in order to further reduce the occlusion, danger detection, danger determination processing, and the like may be performed using images captured by a plurality of cameras.

The imaging unit is preferably installed at a fixed position, but may be installed at a variable position.

Further, in the above embodiment, the case where the safety area (or the danger area) is dynamically changed according to the operating state of the robot arm Rbt_arm and projected onto the projection surface (floor surface FLR) has been described. It is not limited, and for example, the maximum area of the safety area may be detected and the maximum area may be statically displayed on the projection surface (floor surface FLR). In this case, a physically recognizable boundary line or the like may be displayed on the floor surface FLR (for example, the boundary may be emitted by an optical fiber so that a worker or the like can easily recognize it). ..

Further, in the above embodiment, the case where the projection unit Prj1 uses the projector device has been described, but the present invention is not limited to this, and for example, an LED scanner, a laser scanner, or the like may be used as the projection unit Prj1.

Further, in the above embodiment, the case where the number of projection units Prj1 is one has been described, but the present invention is not limited to this, and the display control system includes a plurality of projection units (for example, a projector device). You may. Then, in the display control system configured as described above, a plurality of projection units (for example, a projector device) may be used to project and clearly indicate a safety area or a danger area without shielding.

The projection unit is preferably installed at a fixed position, but may be installed at a variable position.

Further, in the above embodiment, the case where two points of the pedestal of the robot Rbt are used as reference points in the calibration has been described, but the present invention is not limited to this, and the number of reference points for calibration is 2. The above may be the above, or another position may be used as a reference point.

Further, in the display control system and the display control device described in the above embodiment, each block may be individually integrated into one chip by a semiconductor device such as an LSI, or may be integrated into one chip so as to include a part or all of the blocks. You may.

Although it is referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Further, the method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of the circuit cells inside the LSI may be used.

Further, a part or all of the processing of each functional block of each of the above embodiments may be realized by a program. Then, a part or all of the processing of each functional block of each of the above embodiments is performed by the central processing unit (CPU) in the computer. Further, the program for performing each process is stored in a storage device such as a hard disk or a ROM, and is read and executed in the ROM or the RAM.

Further, each process of the above embodiment may be realized by hardware, or may be realized by software (including the case where it is realized together with an OS (operating system), middleware, or a predetermined library). Further, it may be realized by mixed processing of software and hardware.

For example, when each functional unit of the above embodiment (including a modification) is realized by software, the hardware configuration (for example, CPU, GPU, ROM, RAM, input unit, output unit, etc.) shown in FIG. 21 is busted. (Hardware configuration connected by Bus) may be used to realize each functional unit by software processing.

Further, when each functional unit of the above embodiment is realized by software, the software may be realized by using a single computer having the hardware configuration shown in FIG. 21, or a plurality of computers. It may be realized by the distributed processing using.

Further, the execution order of the processing methods in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed without departing from the gist of the invention.

A computer program that causes a computer to execute the above-mentioned method and a computer-readable recording medium that records the program are included in the scope of the present invention. Here, examples of computer-readable recording media include flexible disks, hard disks, SSDs, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, Blu-ray (registered trademarks), next-generation optical disks, and semiconductors. Memory can be mentioned.

The computer program is not limited to the one recorded on the recording medium, and may be transmitted via a telecommunication line, a wireless or wired communication line, a network typified by the Internet, or the like.

The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the invention.

1000, 1000A, 2000 Display control system Cmr1 Imaging unit

Prj1 Projection unit

100, 100A, 100B

Display control device

5, 5A Prediction processing unit Rbt Robot Rbt_arm Robot arm Rbt_C1, Rbt_C1A Robot control unit

Claims

In a space where a robot arm and a movable object may coexist, a projection surface in which the movable object can be recognized in the safety area, which is an area determined to be safe even if the movable object exists. It is a display control system for displaying.
An imaging unit installed at a position above the robot arm and
(1) In the space, the image captured by the imaging unit from a position above the robot arm, and the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for specifying a safety area when the robot arm is in the predetermined state as teacher data. It is a prediction processing unit that executes prediction processing using the trained model acquired by executing the above, and is an image captured from a position above the robot arm by the imaging unit at the time of prediction processing. By executing the prediction processing using the trained model on the image for prediction processing, a safety area in the space when the image for prediction processing is acquired is predicted, and the predicted safety area is used as the prediction safety area. The prediction processing unit that acquires and generates projected image data based on the prediction safety area,
A projection unit that projects a projection image formed from the projection image data onto the projection surface,
Display control system with.
The prediction processing unit
Based on the control data for controlling the robot arm, training projected image data generated so as to include a plurality of image regions having colors specified according to the degree of safety is projected onto the projection surface. At that time, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit.
The display control system according to claim 1.
The prediction processing unit
Based on the control data for controlling the robot arm, training projected image data generated so as to include a plurality of image regions having brightness specified according to the degree of safety is projected onto the projection surface. At that time, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit.
The display control system according to claim 1.
The prediction processing unit
A plurality of image regions having colors specified according to the degree of safety are generated, and the projected image data is generated so that the image formed by the projected image data includes the generated plurality of image regions. ,
The display control system according to any one of claims 1 to 3.
The prediction processing unit
A plurality of image regions having brightness specified according to the degree of safety are generated, and the projected image data is generated so that the image formed by the projected image data includes the generated plurality of image regions. ,
The display control system according to any one of claims 1 to 3.
In a space where a robot arm and a movable object may coexist, a projection surface in which the movable object can be recognized in the dangerous area, which is an area determined to be dangerous if the movable object exists. It is a display control system for displaying.
An imaging unit installed at a position above the robot arm and
(1) In the space, the image captured by the imaging unit from a position above the robot arm, and the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for identifying a dangerous area when the robot arm is in the predetermined state as teacher data. It is a prediction processing unit that executes prediction processing using the trained model acquired by executing the above, and is an image captured from a position above the robot arm by the imaging unit at the time of prediction processing. By executing the prediction processing using the trained model on the image for prediction processing, the danger area in the space when the image for prediction processing is acquired is predicted, and the predicted danger area is used as the prediction danger area. The prediction processing unit that acquires and generates projected image data based on the prediction danger region,
A projection unit that projects a projection image formed from the projection image data onto the projection surface,
Display control system with.
The prediction processing unit
Based on the control data for controlling the robot arm, training projected image data generated so as to include a plurality of image regions having colors specified according to the degree of danger is projected onto the projection surface. At that time, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit.
The display control system according to claim 6.
The prediction processing unit
Based on the control data for controlling the robot arm, training projected image data generated so as to include a plurality of image regions having brightness specified according to the degree of danger is projected onto the projection surface. At that time, the prediction process is executed using the trained model obtained by executing the learning process using the image captured by the imaging unit.
The display control system according to claim 6.
The prediction processing unit
A plurality of image regions having colors specified according to the degree of danger are generated, and the projected image data is generated so that the image formed by the projected image data includes the generated plurality of image regions. ,
The display control system according to any one of claims 6 to 8.
The prediction processing unit
A plurality of image regions having brightness specified according to the degree of danger are generated, and the projected image data is generated so that the image formed by the projected image data includes the generated plurality of image regions. ,
The display control system according to any one of claims 6 to 8.
The space has a floor surface and
The projection surface is a floor surface in the space.
The display control system according to any one of claims 1 to 10.
The space has a ceiling surface and
The imaging unit is installed on the ceiling surface of the space.
The display control system according to any one of claims 1 to 11.
A robot arm control unit for controlling the robot arm is further provided.
When the prediction processing unit determines that the movable object is likely to move out of the safe area or the movable object is likely to move into the dangerous area, the robot arm control unit determines that the movable object is likely to move out of the safe area. Stop the movement of the robot arm and / or execute the process to generate a warning,
The display control system according to any one of claims 1 to 12.
In a space where robot arms and moving objects may coexist
An imaging unit installed at a position above the robot arm and
A projection unit that projects an image onto a predetermined projection surface,
It is a display control method used in a display control system including.
A display control method for displaying a safety area, which is an area determined to be safe even if a movable object exists, on a projection surface on which a movable object can be recognized in the space.
(1) In the space, the image captured by the imaging unit from a position above the robot arm, and the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for specifying a safety area when the robot arm is in the predetermined state as teacher data. This is a prediction processing step of executing prediction processing using the trained model acquired by executing the above, and is an image captured from a position above the robot arm by the imaging unit at the time of prediction processing. By executing the prediction processing using the trained model on the image for prediction processing, a safety area in the space when the image for prediction processing is acquired is predicted, and the predicted safety area is used as the prediction safety area. The prediction processing step to acquire and generate projected image data based on the prediction safety area.
A projection step of projecting a projection image formed from the projection image data onto the projection surface,
Display control method including.
In a space where robot arms and moving objects may coexist
An imaging unit installed at a position above the robot arm and
A projection unit that projects an image onto a predetermined projection surface,
It is a display control method used in a display control system including.
A display control method for displaying a dangerous area, which is an area determined to be dangerous when a movable object is present, on a projection surface in which the movable object can be recognized in the space.
(1) In the space, the image captured by the imaging unit from a position above the robot arm, and the image captured when the robot arm is in a predetermined state, or the robot arm Learning processing using data including control data for controlling the robot arm so as to be in a predetermined state and (2) information for identifying a dangerous area when the robot arm is in the predetermined state as teacher data. This is a prediction processing step of executing prediction processing using the trained model acquired by executing the above, and is an image captured from a position above the robot arm by the imaging unit at the time of prediction processing. By executing the prediction processing using the trained model on the image for prediction processing, the danger area in the space when the image for prediction processing is acquired is predicted, and the predicted danger area is used as the prediction danger area. The prediction processing step to acquire and generate projected image data based on the prediction danger region.
A projection step of projecting a projection image formed from the projection image data onto the projection surface,
Display control method including.