JP2016197393A - Information processor, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of reducing the possibility of collision of a robot in interference confirmation and improving the efficiency in the checking work of points having a possibility of interference.SOLUTION: The information processor includes: control means that controls motions of a first robot; acquisition means that acquires a viewing position and a posture of an observer; generation means that identifies a field of view of the observer including the first robot based on the viewing position and the posture of the observer and a piece of internal parameter of a display device, and generates a motion image of a second robot based on a piece of motion data of the second robot which may interfere with the first robot, and displays the same on a specific field of view; and output means that outputs motion images on a display unit.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program related to robot control.

  Automatic assembly of products is performed by industrial robots with multiple arms. In this case, it is necessary to teach the operation of the robot so as to avoid collisions between objects such as arm-to-arm collisions and workpieces existing in the work area. In particular, it is very troublesome to adjust the timing so as to avoid collision between arms. Therefore, there is a technique for teaching a robot operation while confirming interference by simulation using a three-dimensional model of a robot or a workpiece (Patent Document 1). However, there is a difference between the simulation and the actual three-dimensional shape and motion of the robot, and interference cannot be confirmed accurately by simulation alone. Therefore, finally, it is necessary to check the interference by executing control data on the actual robot.

JP-A-60-217410

However, there is a high possibility that the robot will collide if the control data of the robot with insufficient interference confirmation is reproduced on the actual machine. In addition, when checking with the actual machine whether the arms are interfering in a situation where multiple arms intersect with each other with a slight difference, the actual robot's machine will be in the way and the area where there is a possibility of interference will come into contact. It is difficult to check whether or not.
It is an object of the present invention to reduce the possibility of a robot colliding in interference confirmation and improve the efficiency of confirmation work at a place where there is a possibility of interference.

  The information processing apparatus according to the present invention includes a control unit that controls the operation of the first robot, an acquisition unit that acquires the viewpoint position and orientation of the observer, the viewpoint position and orientation of the observer, and an internal parameter of the display device. The visual field of the observer including the first robot is identified based on the first robot, and the visual field is displayed in the identified visual field based on motion data of a second robot that may interfere with the first robot. A generating unit configured to generate an operation image of the second robot; and a display unit configured to display the operation image on the display device.

  According to the present invention, it is possible to reduce the possibility of a robot colliding in interference confirmation, and improve the efficiency of confirmation work at a place where there is a possibility of interference.

It is a figure which shows an example of the system configuration | structure of a display system. It is a figure which shows an example of the hardware constitutions of information processing apparatus. It is a figure which shows an example of the hardware constitutions of a display apparatus. 3 is a diagram illustrating an example of a software configuration of the information processing apparatus according to the first embodiment. FIG. It is a figure which shows an example of the software configuration of a display apparatus. 3 is a flowchart illustrating an example of information processing of the information processing apparatus according to the first embodiment. 3 is a diagram illustrating an example of a software configuration of an information processing apparatus according to a second embodiment. FIG. 10 is a flowchart illustrating an example of information processing of the information processing apparatus according to the second embodiment. FIG. 10 is a diagram illustrating an example of a software configuration of an information processing apparatus according to a third embodiment. 10 is a flowchart illustrating an example of information processing of the information processing apparatus according to the third embodiment. FIG. 10 is a diagram illustrating an example of a software configuration of an information processing apparatus according to a fourth embodiment. 10 is a flowchart illustrating an example of information processing of the information processing apparatus according to the fourth embodiment. FIG. 10 is a diagram illustrating an example of a software configuration of an information processing apparatus according to a fifth embodiment. 14 is a flowchart illustrating an example of information processing of the information processing apparatus according to the fifth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<< Embodiment 1 >>
<Overview>
In the first embodiment, a method for examining the interference of a robot having two arms while reducing the possibility that the arms collide with each other will be described. FIG. 1 is a diagram for explaining an overview of a display system. As shown in FIG. 1, in the display system, a CG image 1053 reproduced by CG (Computer Graphics) is superimposed on the observer 1001 by superimposing the operation of the second robot arm 1004 on the state where the first robot arm 1003 operates. Present. Thus, the observer 1001 confirms the interference of the robot arm. Accordingly, it is possible to reduce the possibility that the robot arms come into contact with each other at the time of interference confirmation and the robot is broken. Furthermore, it is possible to reduce the fact that one arm is physically obstructing and obstructing the visual recognition of a portion where there is a possibility of interference with the other arm, so that work efficiency is improved. The CG image 1053 is an example of an operation image. Further, the first robot arm 1003 of this embodiment is an example of a first robot. Further, the second robot arm 1004 of the present embodiment is an example of a second robot. Further, the second robot arm 1004 of the present embodiment is an example of a robot arm that may interfere with the first robot arm 1003 of the present embodiment.

<System configuration>
The configuration of the first embodiment will be described with reference to FIG. As shown in FIG. 1, the display system includes an information processing apparatus 1, a first robot arm 1003, a second robot arm 1004, and a sensor 1013 that measures the viewpoint position / posture of the observer 1001. Including. Further, the display system includes a display device (HMD: Head Mounted Display) 1051 in which a camera and a liquid crystal display are incorporated in portions corresponding to the left and right viewpoint positions of the observer 1001. An observer 1001 wears a sensor 1013 and a display device 1051. The display device 1051 is disposed at the viewpoint of the observer 1001, for example. The object 1006 is on the workspace 1005. The scene 1052 is a scene observed by the observer 1001. The CG image 1053 is a CG image of the operation of the second robot arm 1004.
The information processing apparatus 1 controls the movement of the first robot arm 1003 and calculates the viewpoint position and orientation of the observer 1001 based on the signal from the sensor 1013. Then, the information processing apparatus 1 identifies the visual field of the observer from the viewpoint position and orientation of the observer 1001 and the internal parameters of the display device 1051. Here, the information on the display device 1051 is a parameter representing a projection model of the display device 1051 and includes information such as a focal length, a resolution, and a pixel pitch. The information processing apparatus 1 generates a CG image of the operation of the second robot arm 1004 in synchronization with the actual operation of the first robot arm 1003 at the display position of the second robot arm 1004 in the specified visual field. Then, the information processing apparatus 1 generates a superimposed image in which the CG image is superimposed on a video acquired from a camera built in the display apparatus 1051 and transmits the superimposed image to the display apparatus 1051. The display device 1051 displays the superimposed image on a liquid crystal display. Here, it is assumed that calibration between the viewpoint position and orientation of the observer 1001 and the display device 1051 is performed in accordance with the type of the display device 1051 and the like.
In the present embodiment and the embodiments described below, the robot may take any form as long as it has one or more operating parts. Any number of arms may be used as long as the arm or leg can be operated, and a plurality of robots may be used. The robot may be a 6-axis robot, a SCARA robot, or a parallel link robot, and the form is not limited.

<Hardware configuration>
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus 1. The information processing apparatus 1 includes a CPU 11, a ROM 12, a RAM 13, a communication I / F 14, a display device 15, an input device 16, and an HDD 17 as hardware configurations. The CPU 11 controls various devices of the information processing apparatus 1 connected to the system bus. The ROM 12 stores a BIOS program and a boot program. The RAM 13 is used as a main storage device for the CPU 11. The communication I / F 14 connects the information processing apparatus 1 to a network and controls information communication via the network. The network may be connected via a wired connection, may be connected via a wireless connection, or may be connected via a wired or wireless connection. The display device 15 displays the processing result in the CPU 11 or the like. The input device 16 receives input from the operator. The input device 16 may be, for example, a mouse or a keyboard, or a remote controller described later. The HDD 17 stores OS programs, programs of various applications that operate on the OS, and the like.
In the above configuration, when the power of the information processing apparatus 1 is turned on, the CPU 11 reads the OS program from the HDD 17 into the RAM 13 according to the boot program stored in the ROM 12 and executes the process, thereby executing the processing of the information processing apparatus 1. Realize the function. That is, when the CPU 11 of the information processing apparatus 1 executes processing based on the program, the software configuration of the information processing apparatus 1 and the processing of the flowchart described later are realized.

FIG. 3 is a diagram illustrating an example of a hardware configuration of the display device 1051. The display device 1051 includes a CPU 21, a ROM 22, a RAM 23, a communication I / F 24, a display device 25, an input device 26, and a photographing device 27 as hardware configurations. The CPU 21 controls various devices of the display device 1051 connected to the system bus. The ROM 22 stores an OS program, various applications that operate on the OS, and the like. The RAM 23 is used as a main storage device for the CPU 21. The communication I / F 24 connects the display device 1051 to a network and controls information communication via the network. The network may be connected via a wired connection, may be connected via a wireless connection, or may be connected via a wired or wireless connection. The display device 25 is, for example, a liquid crystal display or the like, and displays a CG image or the like generated by the information processing device 1. The input device 26 receives input from the operator. The imaging device 27 captures an image of the real space.
In the above configuration, when the power of the display device 1051 is turned on, the CPU 21 implements the function of the display device 1051 by reading the program stored in the ROM 22 into the RAM 23 and executing the processing. That is, the CPU 21 of the display device 1051 executes processing based on the program, thereby realizing the software configuration of the display device 1051 and the like.

<Software configuration>
FIG. 4 is a diagram illustrating an example of a software configuration of the information processing apparatus 1. As illustrated in FIG. 4, the information processing apparatus 1 includes, as a software configuration, a data holding unit 100, a setting unit 101, an image acquisition unit 102, a viewpoint position / orientation acquisition unit 103, a superimposed image generation unit 104, An output unit 105 and a robot control unit 106 are included.
The data holding unit 100 holds the three-dimensional shape information of the work environment including the robot, the control data of the robot, and the data indicating the operation of the robot (hereinafter referred to as operation data) in the HDD 17 or the like. The three-dimensional shape information of the work environment including the robot includes a three-dimensional mesh model of the work environment including the robot and the joint information of the robot. However, the three-dimensional shape information may be an analytical curved surface model such as a CAD model. The control data of the robot includes a list of data describing actions such as the position of the via point, the joint angle of the robot at the via point, and the opening and closing of the end effector. However, the robot control data may be a list of end effector position and orientation data and action data at the waypoints. The robot control data may be in a text format written in a language for the robot such as a script or a programming language, or may be a set of signals transmitted to the robot. The robot control data may be any data as long as it can reproduce the motion of the robot.
The motion data is data representing the motion of the robot obtained by measuring the actual motion of the robot in advance, and includes a list of data describing actions such as joint angles of the robot and opening / closing of the end effector in time series. The motion of the robot can be acquired using a known motion capture technique. However, the operation data may be any data as long as it represents the operation of the robot. The motion data may be an arbitrary viewpoint image group or a three-dimensional measurement data group constructed by photographing the robot in time series with a plurality of imaging devices or three-dimensional measurement devices.

The setting unit 101 sets parameters such as the robot arm to be drawn by the superimposed image generation unit 104 and the playback speed and playback position of the robot operation. In the present embodiment, the setting unit 101 sets playback information such as an arm to be displayed, a playback speed, and a playback position in response to a setting operation via an input device similar to a remote controller of a video playback device such as a DVD player. More specifically, the setting unit 101 sets parameters for starting the reproduction of the motion of the actual robot and the superimposed image according to the selection operation of the reproduction button. Further, in response to the stop button selection operation, the setting unit 101 sets a parameter for stopping reproduction. Further, in response to the selection operation of the slow playback button or the fast forward button, the setting unit 101 sets parameters to be played by changing the operation speed. Further, in accordance with the channel button selection operation, the setting unit 101 may set a parameter for selecting an arm to be virtually displayed or an arm to be operated on an actual machine. The input device used for setting is not limited to a remote controller, and may be a mouse, a keyboard, or a touch panel. The setting unit 101 may set parameters and the like through buttons, operation sequences, and cursors displayed on the display device 15. The information processing apparatus 1 may sense the movement of the observer 1001 and the setting unit 101 may set parameters and the like according to a gesture performed by the observer 1001. The set parameters or the like may be displayed on the display device 15 or a display of the input device 16 such as a remote controller. Further, the setting unit 101 may automatically determine the arm for virtual display or actual machine operation according to a predetermined rule. For example, the setting unit 101 may determine an arm that performs virtual display or actual machine operation in the order of the ID of each arm.
The image acquisition unit 102 acquires a video to be presented to the observer 1001 from a camera built in the display device 1051 in accordance with the viewpoint position and orientation acquisition unit 103 acquiring the viewpoint position and orientation of the observer 1001. The image acquired by the image acquisition unit 102 is an example of a captured image.
The viewpoint position and orientation acquisition unit 103 acquires the viewpoint position and orientation of the observer 1001. The sensor 1013 shown in FIG. 1 is a 6-degree-of-freedom sensor that can measure the position and orientation by measuring the magnetic force generated from the magnetic field generator. The viewpoint position and orientation acquisition unit 103 acquires the viewpoint position and orientation of the observer 1001 based on information from the sensor 1013.

The superimposed image generation unit 104 is based on the viewpoint position and orientation acquired by the viewpoint position / orientation acquisition unit 103, the three-dimensional shape information and operation data held by the data holding unit 100, and the parameters set by the setting unit 101. The following image is generated. In other words, the superimposed image generation unit 104 identifies the field of view of the observer 1001 based on the viewpoint position and orientation and the angle of view information of the display device 1051, and sets the second robot arm 1004 of the identified field of view to the display position of the second robot arm 1004. An image that reproduces the behavior of the second robot arm 1004 is generated. Then, the superimposed image generation unit 104 generates an image in which an image obtained by reproducing the motion of the robot arm is superimposed on the image acquired by the image acquisition unit 102. In the present embodiment, the superimposed image generation unit 104 reproduces the motion data, determines a reproduction position (time) based on the reproduction information set by the setting unit 101, and the second robot arm 1004 at the reproduction position. Draw the state. The superimposed image generation unit 104 uses the viewpoint position and orientation acquired by the viewpoint position and orientation acquisition unit 103 as the drawing viewpoint. The superimposed image generation unit 104 uses a mesh model included in the three-dimensional shape information for the robot shape. The angle of view information is an example of an internal parameter of the display device.
The output unit 105 transmits the image generated by the superimposed image generation unit 104 to the display device 1051.
The robot control unit 106 operates the first robot arm 1003 based on the robot control data held by the data holding unit 100 and the parameters set by the setting unit 101. The robot control unit 106 reproduces the control data and sets the first robot arm 1003 to be in the state of the robot arm at the reproduction position determined by the superimposed image generation unit 104 based on the parameters set by the setting unit 101. Control parameters such as joint angles. When the playback position is between control data via points, the robot control unit 106 controls the position and orientation of the robot with a value obtained by interpolating parameters at the front and rear via points.
In the present embodiment, the configuration in FIG. 3 has been described as a software configuration. However, the entire configuration or a part of the configuration in FIG. 3 may be implemented in the information processing apparatus 1 as a hardware configuration. The same applies to the following embodiments.

FIG. 5 is a diagram illustrating an example of a software configuration of the display device 1051. As illustrated in FIG. 5, the display device 1051 includes a reception unit 111 and a display unit 112 as a software configuration.
The receiving unit 111 receives an image transmitted from the information processing apparatus 1. The display unit 112 displays the image received by the receiving unit 111 on the display device 25.
In the present embodiment, the configuration in FIG. 5 has been described as a software configuration, but the entire configuration in FIG. 5 or a part of the configuration in FIG. 5 may be implemented in the display device 1051 as a hardware configuration. The same applies to the following embodiments.

<Process flow>
The flow of information processing of the information processing apparatus 1 will be described using the flowchart of FIG.
In step S <b> 1010, the superimposed image generation unit 104 acquires the three-dimensional shape information, control data, and operation data held by the data holding unit 100.
In step S <b> 1020, the setting unit 101 selects parameters such as a robot arm to be drawn by the superimposed image generation unit 104 (and a robot arm that is actually operated by the robot control unit 106), a reproduction speed and a reproduction position of the robot. The observer 1001 may advance the playback position by frame advance, or may continuously operate by specifying a playback speed such as fast-forward or slow playback. The observer 1001 may observe the interference from various viewpoints by temporarily stopping the reproduction position where interference confirmation is difficult.
In step S <b> 1030, the image acquisition unit 102 captures an image of the real space that is presented to the viewer 1001 via the display device 1051.
In S1040, the viewpoint position and orientation acquisition unit 103 acquires the viewpoint position and orientation of the observer 1001 for generating a superimposed image via the sensor 1013.
In S1050, the superimposed image generation unit 104 identifies the visual field of the observer 1001 based on the information acquired or set in S1010 to S1040, and generates a superimposed image to be presented to the observer 1001. The superimposed image generation unit 104 outputs a superimposed image to the output unit 105 and outputs trigger information for operating the first robot arm 1003 to the robot control unit 106.
In S1060, the output unit 105 transmits the image generated in S1040 to the display device 1051, for example, via wireless.

In S1070, the robot control unit 106 operates the first robot arm 1003 based on the three-dimensional shape information acquired in S1010, the control data, the parameters set in S1020, and the trigger information output in S1050. Thereby, the first robot arm 1003 can be moved in synchronization with the second robot arm 1004 drawn in the superimposed image. In this embodiment, the start timing of the operation of the first robot arm 1003 is controlled by the trigger from the superimposed image generation unit 104, but the present invention is not limited to this. For example, it may be controlled by a controller held by the observer.
In S1080, the setting unit 101 determines whether or not the interference check has been completed based on the selection operation or the like of the observer 1001. If the setting unit 101 determines that the interference check has not been completed, the process returns to S1020. If the setting unit 101 determines that the interference check has been completed, the setting unit 101 ends the process of the flowchart illustrated in FIG. When the processing returns to S1020, the observer 1001 performs interference confirmation by observing from various viewpoints while changing the viewpoint position and posture and the reproduction position of the robot operation. The observer 1001 observes an image in which the CG image of the second robot arm 1004 is superimposed on the image of the first robot arm 1003 taken by the HMD built-in camera. Interference with the real environment can be confirmed.

<Variation>
In this embodiment, an example in which the interference of one arm of a robot having two arms is confirmed has been described. However, the present invention is not limited to the above example, and the arm that is virtually displayed on the CG and the arm that is actually operated may be switched to further confirm interference. Further, even in a robot having three or more arms, interference can be confirmed in the same manner by switching the virtual display arm and the arm to be actually operated.
The robot may be in any form as long as it has one or more operating parts. It is sufficient if there is a part that works with both arms and legs. The robot may be a 6-axis robot, a SCARA robot, a parallel link robot, or a combination demonstration of a plurality of robots.
The data holding unit 100 only needs to hold the three-dimensional shape information of the work environment including the robot, the control data of the robot, and data representing the operation of the robot. The three-dimensional shape information may be an analytical curved surface model such as a CAD model. The control data may be a list of end effector position / posture and action data at the via points. The control data may be in a text format written in a language for the robot such as a script or a programming language, or may be a set of signals transmitted to the robot. The control data may be any information as long as it can reproduce the robot operation. The motion data may be data representing the motion of the robot. The motion data is not a list of actually measured parameters such as joint angles of the robot, but is an arbitrary viewpoint image group or three-dimensional measurement data group constructed by photographing the robot in time series with a plurality of imaging devices or three-dimensional measurement devices. There may be. The data held by the data holding unit 100 has been described as being stored in the HDD 17. However, for example, the data may be stored in a memory of another device that can communicate with the information processing device 1 via a network. Further, when confirming the interference, the actual machine of one arm for virtual display need not be arranged.

  The setting unit 101 may set parameters and the like according to the operation through any input device as long as the reproduction information of the robot operation can be set. The reproduction information includes information such as a virtual display, an arm for operating the real machine, a reproduction speed, and a reproduction position. The setting method by the observer 1001 may be set using a button on the remote controller, or the playback position may be set using a jog dial or the like. The setting unit 101 may display the set reproduction information on the display device 15. Further, the CPU of the input device such as a remote controller may display the reproduction information on the display of the input device. The input device is not limited to a remote controller, and may be a device that captures a mouse, a keyboard, or a gesture, analyzes the gesture, and identifies the meaning of the gesture. The setting unit 101 may set parameters in accordance with buttons, operation sequences displayed on the display device 15, and an observer's setting operation on the cursor. For example, the setting unit 101 uses the input device such as a mouse, a touch panel, or a cursor key from the list of robot arm names displayed on the display device 15 by the observer 1001 to select an arm for virtual display or actual machine operation. You may do it. The setting unit 101 may sense the movement of the observer 1001 and virtually display the arm of the robot touched or pointed by the observer 1001 or may be selected as an arm using an actual machine.

The image acquisition unit 102 only needs to acquire a video to be presented to the observer 1001. However, when an optical see-through display is used as the display device 1051, the image acquisition unit 102 may not be a software configuration of the information processing device 1. A color camera may be used as the imaging device, or a gray scale camera or an infrared camera may be used. The imaging device may be attached to the display device 1051 or may not be attached.
In addition, when a two-dimensional liquid crystal display or the like is used as the display device 1051, an observer 1001 can perform various operations on the display screen without changing his / her position by arranging a camera capable of acquiring a position and orientation at an arbitrary position. Interference check from various positions and directions becomes possible.
The viewpoint position and orientation acquisition unit 103 may use any method as long as it can acquire the three-dimensional position and orientation of the viewpoint of the observer 1001. The viewpoint position / orientation acquisition unit 103 may use motion capture technology, or may estimate the viewpoint position / orientation from the measurement values of the GPS, the attitude sensor, the acceleration sensor, etc. of the apparatus worn by the observer 1001. good. The viewpoint position / orientation acquisition unit 103 acquires the three-dimensional position / orientation of the viewpoint of the observer 1001 by a method using an image marker or a method of tracking a characteristic point extracted from an image acquired by the image acquisition unit 102. You may do it. Further, the viewpoint position / orientation acquisition unit 103 may read the image and sensor data for calculating the viewpoint position, attitude, viewpoint position, and attitude from the HDD 17 and acquire the viewpoint position and attitude.

  Based on the viewpoint position and orientation acquired by the viewpoint position and orientation acquisition unit 103, the superimposed image generation unit 104 combines the coordinates of the real environment and the coordinates of the robot to be virtually displayed, and the robot at the set reproduction position (time). An image in which the arm moves is generated. The playback position may be a time when playback is performed at the same speed as when the operation data is acquired, or may be a time when playback is performed slowly or quickly at a set magnification. The playback position may be an arbitrary playback position set by the setting unit 101. The image to be superimposed may be an image drawn with polygons from 3D shape information such as a robot or a workpiece, or may be an image drawn with a wire frame. May be. When an optical see-through display unit is used, the superimposed image generation unit 104 may generate an image of how the robot operates. In addition, when a video see-through display unit is used, the superimposed image generation unit 104 may generate an image in which an image of a robot operating is superimposed on an image captured by the imaging unit. Information regarding whether the display device 1051 is an optical see-through type or a video see-through type may be acquired by the information processing device 1 from the display device 1051 via wireless or the like, for example, or in accordance with the operation of the observer 1001 101 may acquire. For example, the superimposed image generation unit 104 generates a corresponding superimposed image according to information on whether the display device 1051 is an optical see-through type or a video see-through type.

The output unit 105 may be anything as long as it can output the superimposed image to the display device 1051. For example, the output unit 105 transmits the superimposed image to the display device 1051 via wireless or the like, and causes the display device 1051 to display the superimposed image. The display device 1051 may be an optical see-through type or a video see-through type. The display device 1051 may be an HMD or a display attached to a display, a projector, a tablet terminal, a smartphone, or a robot teaching pendant.
The robot control unit 106 only needs to be able to move the robot arm to the position and orientation of the robot arm when the control data is reproduced at the same reproduction position (time) as the superimposed image generation unit 104. Here, the position and orientation of the arm of the robot is synonymous with the joint angle of each joint of the arm of the robot.

  According to the configuration described above, the information processing apparatus 1 generates the motion of the robot arm as a superimposed image using the actual motion trajectory of the robot arm measured in advance. Then, the superimposed image is superimposed on an image obtained by photographing the other robot arm actually operated. By processing in this way, there is an effect that it is possible to perform an interference check or the like without the thought that the robot arm may collide with the movement of the robot arm that cannot be reproduced by simulation. In addition, since it is possible to reduce the difficulty of confirming interference with the other arm due to physical obstruction of one robot arm, work efficiency is improved.

<< Embodiment 2 >>
<Overview>
In the second embodiment, a case where the operation of a robot having two arms is taught will be described. In this embodiment, the observer 1001 directly teaches the operation of the robot using direct teaching in which the robot's arm is directly moved to input the operation. The observer 1001 teaches one arm at a time. Do not operate the arm of the robot that has been taught. The information processing apparatus according to the present embodiment generates an image so as to superimpose and display the image of the motion simulation of the arm that has been taught in the field of view of the observer 1001, and transmits the generated image to the display device. Thereby, the possibility that the taught arm contacts the observer 1001 and other arms can be reduced. Furthermore, since it is reduced that the taught arm becomes an obstacle and it is difficult to confirm interference, the work efficiency is improved.
<Configuration>
FIG. 7 shows the configuration of the information processing apparatus 2 in the present embodiment. The information processing apparatus 2 has the same hardware configuration as the information processing apparatus 1 of the first embodiment. The robot control unit 206 controls a robot including the first robot arm 1003 and the second robot arm 1004. The data holding unit 200, the setting unit 201, the image acquisition unit 202, the viewpoint position / orientation acquisition unit 203, the superimposed image generation unit 204, and the output unit 205 are the same as the corresponding units in the first embodiment. In addition to these, the information processing apparatus 2 according to the second embodiment includes an operation input unit 207. Below, it demonstrates centering on the difference with Embodiment 1. FIG. The second robot arm 1004 of this embodiment is an example of a third robot. The first robot arm 1003 of this embodiment is an example of a fourth robot. The first robot arm 1003 of this embodiment is an example of a robot arm that may interfere with the second robot arm 1004 of this embodiment.
The motion input unit 207 inputs the data of the robot moved by the observer 1001 directly holding the robot arm by direct teaching to the information processing apparatus 2. More specifically, the motion input unit 207 adds parameters such as joint angles and actions of each joint of the robot after movement to the control data as parameters of waypoints, and causes the data holding unit 200 to hold them.

<Process flow>
The flow of information processing of the information processing apparatus 2 will be described using the flowchart of FIG.
The processes of S2010 to S2040 and S2050 to 2060 are the same as the processes of S1010 to S1040 and S1050 to 1060 of the first embodiment.
In S2045, the motion input unit 207 creates control data of the second robot arm 1004 that causes the actual machine selected in S2020 to operate. While observing the superimposed image of the first robot arm displayed on the display device 1051, the observer 1001 teaches the second robot arm 1004 by direct teaching so that the arm does not interfere with the first robot arm. The motion input unit 207 allows the observer 1001 to directly touch the robot arm based on the position and orientation of the second robot arm 1004 (initial position and orientation of the second robot arm 1004 in the first case) input in step S2045 before one loop. Input data after touching and moving. That is, the motion input unit 207 adds the position and orientation of the second robot arm 1004 after movement (parameters such as joint angles and actions of each joint) to the control data as a via point, and causes the data holding unit 200 to hold it.

In S2050, the superimposed image generation unit 204 determines the viewpoint position and orientation acquired in S2040, the three-dimensional shape information and motion data of the robot acquired in S2010, the parameters set in S2020, and the movement of the robot input in S2045. Based on the information related to the above, an image in which the first robot arm 1003 to be virtually displayed operates is created. More specifically, the superimposed image generation unit 204 simulates an operation of moving the second robot arm 1004 moved in S2045 from the position and orientation in S2045 one loop before to the position and orientation input in S2045 this time. The time required for movement (movement time) is calculated. The superimposed image generation unit 204 determines the first position at the reproduction position advanced by the moving time from the position and orientation of the first robot arm 1003 that is virtually displayed in the superimposed image generated in S2050 one loop before according to the operation data and the like. An image reproducing the operation of the robot arm 1003 is generated.
In step S2080, the motion input unit 207 determines whether teaching of the second robot arm 1004 by direct teaching of the observer 1001 has been completed. For example, the motion input unit 207 determines that teaching has ended when there is no motion input by direct teaching for a predetermined period. The operation input unit 207 ends the process of the flowchart shown in FIG. 8 when it is determined that the teaching is completed, and returns the process to step S2020 when it is determined that the teaching is not ended.

<Variation>
In this embodiment, an example in which the operation of one arm of a robot having two arms is input while confirming interference has been described. However, after the processing in the above example, the observer 1001 may further perform an operation input by exchanging the virtual display arm and the operation input arm. Further, even if the observer 1001 is a robot having three or more arms, the observer 1001 can input the operation in the same manner by switching the virtual display arm and the operation input arm in order.
The motion input unit 207 may be anything as long as it can control the motion of the robot. As described above, the motion input unit 207 may input data of the motion that the observer 1001 has moved directly holding the arm of the robot, or the robot at the waypoint input by the observer 1001 using the teaching pendant. The position, joint angle, action, etc. may be input. Further, the motion input unit 207 may generate robot control data based on the measured motion of the observer 1001. The motion input unit 207 estimates the position and orientation of the end effector and the gripping object from the measurement data obtained by observing the movement of the end effector and the gripping object with a camera and a three-dimensional measurement device, and the robot based on the position and orientation The control data may be created. The observer 1001 may correct the control data by inputting the operation of the robot by the above method when interference is confirmed by the method of the first or second embodiment. For example, the observer 1001 designates a via point of an operation to be corrected, and edits a robot control parameter (joint angle, position, action, etc.) at the via point using a teaching pendant, direct teaching, a computer terminal, or the like. You may do it. The information processing apparatus receives an operation related to editing by an observer 1001 using a teaching pendant, direct teaching, a computer terminal, or the like, and edits control data. The control data to be edited may be for the first robot arm 1003 for virtual display or for the second robot arm 1004 for operating the actual machine.
According to the configuration described above, in the operation teaching of the robot having a plurality of arms, it is possible to reduce the possibility that the taught arm comes into contact with the observer 1001 or other arms. Furthermore, since it is reduced that it is difficult to confirm interference due to the taught arm being in the way, the efficiency of the teaching work is improved.

<< Embodiment 3 >>
<Overview>
In the third embodiment, an example will be described in which, when confirming interference, a portion where interference should be confirmed is emphasized and displayed in a superimposed manner. By adopting such a configuration, the observer can reduce the trouble of confirming the interference of a portion that does not clearly interfere, and the work efficiency is improved.
The configuration of the information processing apparatus 3 in this embodiment is shown in FIG. The information processing apparatus 3 has the same hardware configuration as that of the first embodiment. The robot control unit 306 controls a robot including the first robot arm 1003 and the second robot arm 1004. The data holding unit 300, setting unit 301, image acquisition unit 302, viewpoint position / orientation acquisition unit 303, superimposed image generation unit 304, and output unit 305 are the same as the corresponding units in the first embodiment. In addition to these, the information processing apparatus 3 according to the third embodiment includes an interference determination unit 308. Below, it demonstrates centering on the difference with Embodiment 1. FIG.
The interference determination unit 308 determines whether or not there is a place where the robot arms are likely to interfere with each other, and detects a place where there is a possibility of the interference. More specifically, the interference determination unit 308 reproduces the reproduction position (time) determined from the three-dimensional shape information of the robot arm held by the data holding unit 300, control data, operation data, and parameters set by the setting unit 301. ) To simulate the robot movement. The interference determination unit 308 performs a simulation of a scene at a predetermined reproduction position, and determines that there is a possibility of interference if there is a location that is a distance equal to or less than a preset threshold between the robot arms. Detect interference points.

<Process flow>
The flow of information processing of the information processing apparatus 3 will be described using the flowchart of FIG.
The processes of S3010 to S3040 and S3050 to S3080 are the same as the processes of S1010 to S1040 and S1050 to S1080 of the first embodiment.
In step S3047, the interference determination unit 308 determines whether there is a place where the robot arms are likely to come into contact with each other, and detects a place where there is a possibility of interference. More specifically, the interference determination unit 308 simulates the operation of the second robot arm 1004 that virtually displays based on the three-dimensional shape and operation data held by the data holding unit 300. Further, the interference determination unit 308 performs an interference determination by simulating the operation of the first robot arm 1003 that is actually operated based on the three-dimensional shape and the control data. The interference determination unit 308 performs a simulation of the robot at the reproduction position advanced according to the reproduction speed set by the setting unit 301 from the position and orientation of the robot at the reproduction position one loop before. Then, the interference determination unit 308 determines that there is a possibility of interference if there is a portion having a distance equal to or less than a preset threshold value between the robot arm, the workpiece, the workspace, and the like. To detect.
In S3050, the superimposed image generation unit 304 virtually displays the second robot arm 1004 from the viewpoint position and orientation acquired in S3040, the three-dimensional shape information and motion data acquired in S3010, and the parameters set in S3020. Create an image that works. Furthermore, the superimposed image generation unit 304 generates an image by emphasizing a portion with the possibility of contact detected in S2047. More specifically, the superimposed image generation unit 304 displays the color of the three-dimensional shape model of the robot around the detected location in red.

<Variation>
The interference determination unit 308 only needs to perform a robot motion simulation from the three-dimensional information and motion data of the robot and detect a portion that may cause interference. For example, the interference determination unit 308 may not use the result of the simulation of the first robot arm 1003 that is actually operated as in the present embodiment. The interference determination unit 308 may perform interference determination between the three-dimensional measurement result of the first robot arm 1003 that is actually operated and the simulation result of the second robot arm 1004 that is virtually displayed. For example, the first robot arm 1003 of the actual robot reproduced at a predetermined reproduction position is moved, and the workspace including the robot is three-dimensionally measured using a three-dimensional measuring device installed in the environment. The interference determination unit 308 determines a location where the distance from the second robot arm 1004 to be virtually displayed reproduced based on the acquired 3D measurement data, motion data, and 3D shape information is equal to or less than a predetermined threshold. You may detect as an interference location. In addition, the interference determination standard used by the interference determination unit 308 may be a distance or an inclusion relationship of a three-dimensional shape. In addition, the interference determination unit 308 may notify the user by generating a sound when an interference location is detected.
When the interference location is highlighted, the superimposed image generation unit 304 may generate an image as long as the interference location is known. The superimposed image generation unit 304 may generate an image in which the color of the interference location is changed, or may generate a blinking image. In addition, the superimposed image generation unit 304 may generate an image in which the interference portion is indicated by an arrow or the like, may generate an image surrounded by a translucent sphere or a rectangular parallelepiped, or may display an image displaying a sentence. It may be generated.
Further, as in the second embodiment, an operation input unit may be mounted on the information processing apparatus, and control data may be input or corrected while checking interference.
According to the configuration described above, it is possible to reduce the trouble of confirming the interference at a location that does not clearly interfere, and the work efficiency is improved.

<< Embodiment 4 >>
<Overview>
In the fourth embodiment, an example will be described in which the shape of an actual robot, its peripheral devices, work environment, and the like is different from the three-dimensional shape information for generating a motion image of the robot. In this case, the information processing apparatus three-dimensionally measures the robot, its peripheral devices, the work environment, and the like, and generates an operation image of the robot based on the measured three-dimensional shape information. By adopting such a configuration, it is possible to perform interference confirmation with high accuracy while observing an operation image reflecting the three-dimensional shape of an actual robot, peripheral devices, and work environment, and work efficiency is improved.
An example of the software configuration of the information processing apparatus 4 in this embodiment is shown in FIG. In addition to the hardware configuration similar to that of the first embodiment, the information processing apparatus 4 includes a measurement device (not illustrated) that measures a three-dimensional shape of the robot, its peripheral devices, and the environment. In this embodiment, an RGBD camera (a camera capable of simultaneously capturing a color image and a distance image) will be described as an example of a measurement device. The robot control unit 406 controls a robot including the first robot arm 1003 and the second robot arm 1004. The data holding unit 400, the setting unit 401, the image acquisition unit 402, the viewpoint position / orientation acquisition unit 403, the superimposed image generation unit 404, and the output unit 405 are the same as the corresponding units in the first embodiment. In addition to these, the information processing apparatus 4 includes a shape measuring unit 409 as a software configuration. Below, it demonstrates centering on the difference with Embodiment 1. FIG.
The shape measuring unit 409 measures and acquires three-dimensional shape information of an actual robot, a peripheral device such as a jig used for work, and a work environment using a measuring device. In the present embodiment, the shape measurement unit 409 moves the measurement device, and acquires color images and distance images obtained by measuring the robot, peripheral devices, and work environment from various viewpoints. The information processing apparatus 4 calculates the position and orientation of the photographing viewpoint by extracting and matching features from the acquired color image and distance image, and coordinates the three-dimensional point group generated from each color image and distance image with one coordinate. Integrated into the system and meshed. As a result, three-dimensional shape information of the robot, peripheral devices, and work environment is generated. Further, the information processing apparatus 4 aligns the generated 3D shape information with the 3D shape information held by the data holding unit 400 and holds the generated coordinate system of the 3D shape information in the data holding unit 400. Match the coordinate system of the 3D shape information of the robot. At this time, the information processing apparatus 4 associates and aligns a three-dimensional shape having operation data of a robot or the like for each movable part.

<Process flow>
The flow of information processing of the information processing apparatus 4 will be described using the flowchart of FIG.
The processes of S4010 and S4020 to S4080 are the same as the processes of S1010 and S1020 to S1080 of the first embodiment.
In step S4015, the shape measurement unit 409 acquires three-dimensional shape information obtained by measuring an actual robot, peripheral devices, and work environment. The process of S4015 is an example of a process for acquiring three-dimensional shape information.
In S4050, the superimposed image generation unit 404 creates an image in which the second robot arm 1004 to be virtually displayed is operated from the following data or the like. In other words, the superimposed image generation unit 404 creates an image to be virtually displayed from the viewpoint position and orientation acquired in S4040, the motion data acquired in S4010, the three-dimensional shape information acquired in S4015, and the parameters set in S4020. create.

<Variation>
The shape measuring unit 409 only needs to be able to acquire three-dimensional shape information such as an actual robot, its peripheral devices, and the work environment. The shape measuring unit 409 may measure a three-dimensional shape using an RGBD camera, a ToF sensor, or a stereo camera. In addition, the shape measuring unit 409 may use a combination of three-dimensional shape information measured from a plurality of viewpoints by moving a measuring device or using a plurality of measuring devices. A plurality of measuring devices may calibrate their shooting positions and orientations in advance. The three-dimensional shape information may be a three-dimensional point group, a set of surfels (Surfell: Surface Element), a mesh model, or a CAD model. Texture information may be included. Further, the measured three-dimensional shape information and the three-dimensional shape information held in the data holding unit may be used in combination.
Although the present embodiment has been described with respect to the same configuration as the first embodiment, it may be combined with the second and third embodiments. More specifically, the information processing apparatus 4 performs the second and third embodiments by using the three-dimensional shape information measured by the shape measuring unit 409 instead of the three-dimensional shape information held by the data holding unit. It ’s fine.
The robot control unit 406 controls and operates the robot and peripheral devices based on the robot control data held by the data holding unit 400 and the parameters set by the setting unit 401. When the robot and peripheral devices operate, if there is a possibility of contact with the work environment measured by the shape measuring unit 409, for example, the robot control unit 406 may operate the robot and peripheral devices to avoid contact. Data may be changed.

  According to the configuration described above, the information processing apparatus 4 generates the operation of the robot arm and the peripheral device as a superimposed image using the actually measured three-dimensional shape information of the robot, the peripheral device, and the work environment. Then, the information processing device 4 superimposes the superimposed image on an image obtained by photographing the other robot arm actually operated. By processing in this way, three-dimensional shapes such as robots, peripheral devices, and work environments that could not be reproduced in advance simulation can be reflected in the simulation and superimposed images, and an accurate interference check can be performed. Work efficiency is improved.

<< Embodiment 5 >>
<Overview>
In some cases, the shape and operation of an actual robot, its peripheral devices, work environment, and the like are different from the three-dimensional shape information and operation for generating an operation image of the robot. Therefore, in the fifth embodiment, the information processing apparatus measures the three-dimensional shape and motion of the robot, its peripheral devices, work environment, and the like, and generates a robot motion image based on the measured three-dimensional shape information and motion. By adopting such a configuration, it is possible to perform interference confirmation with high accuracy while observing an operation image reflecting the three-dimensional shape and operation of an actual robot, peripheral devices, and work environment, and work efficiency is improved. .
An example of the software configuration of the information processing apparatus 5 in this embodiment is shown in FIG. The information processing apparatus 5 has the same hardware configuration as that of the fourth embodiment. The robot control unit 506 controls a robot including the first robot arm 1003 and the second robot arm 1004. The data holding unit 500, setting unit 501, image acquisition unit 502, viewpoint position / orientation acquisition unit 503, superimposed image generation unit 504, and output unit 505 are the same as corresponding units in the fourth embodiment. In addition to these, the information processing apparatus 5 includes a shape / motion measuring unit 510 as a software configuration. Below, it demonstrates centering on the difference with Embodiment 4. FIG.
The shape / motion measuring unit 510 in the present embodiment uses an unillustrated measuring device to measure an actual robot, a cable, peripheral equipment such as a jig used for work, three-dimensional shape information and motion information of a work environment, get. In the present embodiment, a plurality of RGBD cameras will be described as an example of the measurement device. The relationship between the positions and orientations of a plurality of RGBD cameras is calibrated in advance and made known. The plurality of RGBD cameras measure the robot, peripheral devices, and work environment from a plurality of viewpoints in synchronization.
The robot control unit 506 controls and operates peripheral devices such as robots and jigs based on the operation data held by the data holding unit 500 when measuring. The information processing device 5 generates a colored three-dimensional point group from the color image and the distance image obtained by measurement. Since each measurement viewpoint position is known by pre-calibration, the information processing apparatus 5 sequentially acquires a three-dimensional point group obtained by integrating the three-dimensional point groups as three-dimensional shape information. This is sequentially repeated until a series of operations are completed, and the information processing apparatus 5 acquires the three-dimensional shape information arranged in time series as the three-dimensional shape information and the operation information.

<Process flow>
The flow of information processing of the information processing apparatus 5 will be described using the flowchart of FIG.
The processes of S5010 and S5020 to S5080 are the same as the processes of S1010 and S1020 to S1080 of the first embodiment.
In step S5013, the setting unit 501 sets parameters such as the playback speed and playback position of the robot.
In step S5015, the shape / motion measurement unit 510 sequentially acquires three-dimensional shape information obtained by measuring an actual robot, peripheral devices, and work environment. The processing of S5015 is an example of shape / motion acquisition processing.
In S5016, the robot control unit 506 controls and operates the robot and peripheral devices such as jigs used for work based on the three-dimensional shape information acquired in S5010, the control data, and the parameters set in S5013.
In step S5017, the shape / motion measurement unit 510 determines whether or not the reproduction of the motion of the series of robots and peripheral devices has been completed. If it is determined that the reproduction has not been completed, the shape / motion measuring unit 510 returns the process to S5015, and if it is determined that the interference check has been completed, the process proceeds to S5018.
In step S5018, the shape / motion measurement unit 510 converts the three-dimensional shape information sequentially acquired in step S5015 into a three-dimensional moving image, and holds it as robot motion data. The shape / motion measuring unit 510 may store a three-dimensional moving image held as robot motion data in the data holding unit 500.
In S5050, the superimposed image generation unit 504 creates an image. In other words, the superimposed image generation unit 504 virtually displays the second position for virtual display from the viewpoint position and orientation acquired in S5040, the motion data acquired in S5010, the three-dimensional video acquired in S5018, and the parameters set in S5020. An image in which the robot arm 1004 operates is created.

<Variation>
The shape / motion measurement unit 510 only needs to acquire three-dimensional shape information and motion such as an actual robot, its peripheral devices, and a work environment. The shape / motion measuring unit 510 may measure a three-dimensional shape using an RGBD camera, a ToF sensor, or a stereo camera. In addition, the shape / motion measurement unit 510 may use a plurality of measurement devices in combination with three-dimensional shape information measured from a plurality of viewpoints. A plurality of measuring devices may calibrate their shooting positions and orientations in advance. The three-dimensional shape information may be a three-dimensional point group, a set of surfels (Surfell: Surface Element), a mesh model, or a CAD model. Texture information may be included. Further, the measured three-dimensional shape information and the three-dimensional shape information held in the data holding unit may be used in combination. The motion data may be a three-dimensional moving image in which the three-dimensional shape information is arranged in time series. The three-dimensional shape information arranged in time series acquired by the shape / motion measuring unit 510 is applied to the three-dimensional shape information (for example, the three-dimensional shape model and joint information of the robot) held by the data holding unit, and is used as the robot control data. It may be converted. Then, the superimposed image generation unit 504 generates an image in which the second robot arm 1004 to be virtually displayed operates based on the three-dimensional shape information acquired by the shape / motion measurement unit 510 and the converted control data. good.
The robot control unit 506 controls and operates the robot and peripheral devices based on the robot control data held by the data holding unit 500 and the parameters set by the setting unit 501. The playback speed of the robot and peripheral devices may be changed by the setting unit 501. Further, when the robot and peripheral devices operate, if there is a possibility of contact with the work environment measured by the shape / motion measurement unit 510, for example, the robot control unit 406 may be configured to avoid contact with the work environment. The operation data of the robot and peripheral devices may be changed. Then, the changed motion may be measured by the shape / motion measuring unit 510.
Although the present embodiment has been described with respect to the same configuration as the first embodiment, it may be combined with the second and third embodiments. More specifically, Embodiments 2 and 3 may be implemented using 3D shape information and operations measured by the shape / motion measuring unit 510 instead of 3D shape information held by the data holding unit.

  According to the configuration described above, the information processing apparatus 5 generates the operation of the robot arm and the peripheral device as a superimposed image using the actually measured robot and the peripheral device, and the three-dimensional shape information and the operation information of the work environment. Then, the information processing apparatus 5 superimposes the superimposed image on an image obtained by photographing the other robot arm actually operated. By processing in this way, it is possible to reflect the three-dimensional shape and operation of the robot, peripheral devices, work environment, etc. that could not be reproduced in the previous simulation in the simulation and superimposed image, and to perform accurate interference check, etc. This has the effect of improving work efficiency.

<< Other Embodiments >>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. It can also be realized by a process in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was explained in full detail, this invention is not limited to the specific embodiment which concerns.

  As mentioned above, according to each embodiment mentioned above, when teaching a robot, possibility that a robot will collide can be reduced and the efficiency of teaching work can be improved.

11 CPU, 12 ROM, 13 RAM, 17 HDD

Claims (13)

Control means for controlling the operation of the first robot;
Acquisition means for acquiring the viewpoint position and orientation of the observer;
A second robot that identifies the visual field of the observer including the first robot based on the viewpoint position and orientation of the observer and an internal parameter of the display device, and may interfere with the first robot Generating means for generating an operation image of the second robot to be displayed in the specified visual field based on the operation data of
Output means for outputting the operation image to the display device;
An information processing apparatus. Further comprising shape information acquisition means for acquiring three-dimensional shape information of the second robot and peripheral devices;
The information processing apparatus according to claim 1, wherein the generation unit generates an operation image of the second robot based on the operation data of the second robot and the three-dimensional shape information. It further has a shape / motion acquisition means for acquiring three-dimensional shape information and motion information of the second robot and peripheral devices,
The information processing apparatus according to claim 1, wherein the generation unit generates an operation image of the second robot based on the three-dimensional shape information and operation information.   4. The information processing apparatus according to claim 1, wherein the generation unit generates an operation image of the second robot to be displayed at a position where the second robot is to be arranged in the identified visual field. .   5. The information processing apparatus according to claim 1, wherein the generation unit generates an operation image of the second robot in synchronization with control of the operation of the first robot by the control unit. Further comprising image acquisition means for acquiring a captured image from the viewpoint position and orientation of the observer;
The information processing apparatus according to claim 1, wherein the generation unit generates a superimposed image obtained by superimposing the captured image and an operation image of the second robot. The control means controls the operation of the first robot based on control data,
The information processing apparatus according to claim 1, further comprising a correction unit that corrects the control data. It further has an operation input means for inputting data of actual movement of the second robot,
The information processing apparatus according to claim 1, wherein the generation unit generates an operation image of the second robot using the input data as operation data of the second robot.   The first robot operation simulation and the second robot operation simulation are performed, and whether or not there is a location where the distance between the first robot and the second robot is equal to or less than a set threshold value. The information processing apparatus according to claim 1, further comprising interference determination means for determining.   The information processing apparatus according to claim 9, wherein when the interference determination unit determines that the location is present, the generation unit generates an operation image of the second robot in which the location is highlighted. An information processing method executed by an information processing apparatus,
A control step for controlling the operation of the first robot;
An acquisition step of acquiring the viewpoint position and orientation of the observer;
A second robot that identifies the visual field of the observer including the first robot based on the viewpoint position and orientation of the observer and an internal parameter of the display device, and may interfere with the first robot Generating a motion image of the second robot to be displayed in the specified visual field based on the motion data of
An output step of outputting the operation image to the display device;
An information processing method including: An information processing method executed by a display system,
A control step for controlling the operation of the first robot;
An acquisition step of acquiring the viewpoint position and orientation of the observer;
A second robot that identifies the visual field of the observer including the first robot based on the viewpoint position and orientation of the observer and an internal parameter of the display device, and may interfere with the first robot Generating a motion image of the second robot to be displayed in the specified visual field based on the motion data of
A display step of displaying the operation image on the display device;
An information processing method including:   The program for functioning a computer as each means of the information processing apparatus in any one of Claims 1 thru | or 10.

Images