JP2024044796A - Image Generator - Google Patents

Abstract

[Problem] To provide an image generating device for examining the placement of a robot. [Solution] The image generating device 3 includes an image generating unit 36 that generates a display image of a three-dimensional model of a virtual robot using the relative positional relationship between a reference marker 6 and a display device 4, an output unit 37 that outputs the display image to the display device 4, a registration unit 34 that registers the position and orientation of an end effector of the virtual robot in response to a registration instruction, and a robot positional relationship acquisition unit 35 that acquires a new positional relationship between the reference marker 6 and the virtual robot in response to a movement instruction, and the image generating unit 36 generates a display image of the three-dimensional model of the virtual robot in which the end effector is in the registered position and orientation, and is placed so as to have a new positional relationship with the reference marker 6, in response to the movement instruction. With this configuration, it is possible to examine the placement position of the virtual robot while fixing the position and orientation of the end effector. [Selected Figure] FIG.

Description

The present invention relates to an image generating device that generates a display image for displaying a three-dimensional model of a virtual robot in which an end effector is at a registered position or position and posture.

Conventionally, catalog specifications were used as a reference when considering the placement of robots when introducing them. Additionally, a simulator that can output the optimal placement of robots has also been used (see Patent Document 1). This simulator allows workpieces and peripheral equipment to be installed in addition to the robot, and furthermore, it is possible to consider the optimal placement by taking into account the differences between the real environment and the virtual environment within the simulator.

JP 2005-022062 A

However, when using a simulator, it is necessary to prepare 3D models of the workpiece, peripheral equipment, etc. in addition to the robot, which increases costs. Furthermore, when considering placement, taking into account the error between the real environment and the virtual environment, if there is a larger error than expected, there is a possibility that the robot will interfere with peripheral equipment when operating in the real environment. For this reason, there is a demand for robot placement considerations that are low cost and highly accurate.

The present invention was made in response to the above situation, and aims to provide an image generation device for considering robot placement at low cost and with high accuracy.

In order to achieve the above object, an image generation device according to one aspect of the present invention includes a storage unit that stores a three-dimensional model of a virtual robot corresponding to a real robot, a reference marker that exists in the real environment, and an image generation device that stores an image in the real environment. a marker positional relationship acquisition unit that acquires the relative positional relationship with a display device that is superimposed on an image of the virtual robot or the real environment itself; and a marker positional relationship acquisition unit that acquires the relative positional relationship with a display device that is superimposed on an image of A reception unit that receives posture registration instructions and virtual robot movement instructions, and a virtual robot that is arranged to have a predetermined positional relationship with a reference marker based on the three-dimensional model of the virtual robot and its relative positional relationship. an image generation unit that generates a display image for displaying the three-dimensional model according to an accepted operation, an output unit that outputs the display image to a display device, and a virtual robot when a registration instruction is accepted. A registration unit that registers the position or position and orientation of the end effector in the three-dimensional model; and a robot positional relationship acquisition unit that acquires a new positional relationship between the reference marker and the virtual robot in response to receiving a movement instruction. , the image generation unit is a three-dimensional model of the virtual robot arranged so as to have a new positional relationship with the reference marker when the new positional relationship is acquired by the robot positional relationship acquisition unit, and the end effector is This method generates a display image for displaying a three-dimensional model of a virtual robot at a registered position or position and orientation.

According to an image generating device according to one aspect of the present invention, it is possible to view a display image of a virtual robot in which the placement position of the virtual robot has been changed, together with the real environment, while keeping the position or the position and orientation of the end effector in the three-dimensional model of the virtual robot fixed. This makes it possible to consider the placement of the robot at low cost and with high accuracy.

A schematic diagram showing the configuration of an information processing system according to an embodiment of the present invention A flowchart showing the operation of the image generating apparatus according to the embodiment. Flowchart showing the operation of the image generation device according to the embodiment FIG. 13 shows a virtual robot before it moves in the embodiment. FIG. 13 shows a virtual robot after it has moved in the embodiment. FIG. 13 shows a plurality of candidates for the movement destination of a virtual robot in the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image generation device according to the present invention will be described below using embodiments. Note that in the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and a repeated explanation may be omitted. The image generation device according to the present embodiment registers the position and orientation of an end effector of a virtual robot, and displays a display image that displays a three-dimensional model of the virtual robot whose position and orientation are registered by the end effector at a new arrangement position. is generated.

FIG. 1 is a schematic diagram showing the configuration of an information processing system 100 according to this embodiment. Information processing system 100 according to this embodiment includes an image generation device 3 and a display device 4. Note that the image generation device 3 and the display device 4 may be connected, for example, by wire or wirelessly.

The virtual robot to be operated is constructed from a three-dimensional model existing in the virtual environment, and corresponds to a real robot. That is, the virtual robot is the same as a real robot except that it is constructed from a three-dimensional model, and may have the same size and configuration as the real robot, and has multiple arm joints similar to the real robot. It may be possible to change the angle, etc. An end effector is attached to the tip of the virtual robot. The end effector is the same as the real end effector except that it is constructed from a three-dimensional model, and may, for example, be the same size and configuration as the real end effector, and may be the same as the real end effector. If it has a movable part, it may have a movable part, similar to the end effector in the actual environment. The end effector may be, for example, a welding torch, a hand having a grip for gripping the object to be transported, a hand on which the object to be transported is placed, a tool having functions such as assembly or painting.

The real robot is usually an industrial robot, and may be a manipulator having a plurality of arms (links) connected by joints driven by motors. The real robot may be, for example, a vertically articulated robot or a horizontally articulated robot. Further, the actual robot may be, for example, a transfer robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. Good too. Note that the real robot is, for example, a robot that exists in a real environment. The real environment refers to an environment in real space.

The image generating device 3 registers the position and posture of the end effector of the virtual robot, and when a new positional relationship between the reference marker 6 and the virtual robot is acquired in response to an instruction to move the virtual robot, the image generating device 3 generates a display image for displaying a three-dimensional model of the virtual robot in which the end effector is in the registered position and posture, and in which the virtual robot is positioned so as to have a new positional relationship with the reference marker 6. In this way, the position of the virtual robot can be considered with the position and posture of the hand fixed. For example, if interference occurs with peripheral devices when the hand of the virtual robot is placed in a desired position and posture, a position of the virtual robot that does not cause interference can be considered with the position and posture of the hand fixed. Details of the image generating device 3 will be described later.

The reference marker 6 is a predetermined two-dimensional image. The reference marker 6 may be, for example, an AR marker, a QR code (registered trademark), or any other two-dimensional image with a predetermined shape. The size of the reference marker 6 may be, for example, predetermined. In this embodiment, the case where the reference marker 6 is displayed on a sheet 6a will be mainly described. The reference marker 6 may be printed on, for example, a sheet 6a made of paper or resin.

The predetermined positional relationship between the reference marker 6 and the three-dimensional model of the virtual robot may be, for example, a predetermined positional relationship, or may be a positional relationship that can be changed by a worker or the like who operates the virtual robot. The predetermined positional relationship may be, for example, a positional relationship in which the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6, or a positional relationship in which the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6. In the former case, the image generating device 3 may place the three-dimensional model of the virtual robot at the position of the reference marker 6. Placing the three-dimensional model of the virtual robot at the position of the reference marker 6 may mean that the three-dimensional model of the virtual robot is placed in the virtual space so that a situation in which a real robot is placed at the position of the reference marker 6 is virtually reproduced, and for example, the three-dimensional model of the virtual robot may be placed so that the end face on the base end side of the three-dimensional model of the virtual robot (for example, the mounting surface to the floor surface, etc.) coincides with the surface of the reference marker 6. Placing the three-dimensional model of the virtual robot at a position different from the reference marker 6 may be, for example, that the three-dimensional model of the virtual robot is placed next to the reference marker 6.

The display device 4 displays an image superimposed on an image of the real environment or the real environment itself. That is, the worker who operates the virtual robot can see both the real environment and the virtual environment images by using the display device 4. The display device 4 may be a wearable display device that the worker who operates the virtual robot wears on his head, or may be a display device that is a portable information processing terminal such as a tablet terminal. The wearable display device may be, for example, a head-mounted display. The display device 4 may also have, for example, a transparent display. In this case, the display device 4 displays an image superimposed on the real environment itself. As a wearable display device 4 having a transparent display, for example, HoloLens (registered trademark) is known. Such a display device 4 having a transparent display can also be considered as a display device for realizing mixed reality (MR: Mixed Reality). The display device 4 may also have, for example, a non-transparent display. In this case, the display device 4 displays an image superimposed on an image of the real environment. Therefore, it is preferable that the display device 4 having a non-transparent display has a camera for photographing the real environment or is connected to a camera for photographing the real environment. The image of the real environment captured by the camera is displayed on the non-transmissive display in real time. For example, Oculus Quest is known as a wearable display device 4 having a non-transmissive display. Such a display device 4 having a non-transmissive display can also be considered as a display device for realizing augmented reality (AR). The display device 4, which is a portable information processing terminal such as a tablet terminal, has, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. In this embodiment, the case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

As shown in FIG. 1, the image generation device 3 includes a storage section 31, a marker positional relationship acquisition section 32, a reception section 33, a registration section 34, a robot positional relationship acquisition section 35, and an image generation section 36. , and an output section 37.

The storage unit 31 stores a three-dimensional model of a virtual robot. The three-dimensional model of the virtual robot may correspond to, for example, a real robot that is being considered for introduction. If there are multiple targets to be considered for introduction, for example, three-dimensional models of multiple virtual robots may be stored in the storage unit 31. In this case, a display image may be generated while appropriately switching three-dimensional models of a plurality of virtual robots. It is assumed that an end effector is attached to the hand of the three-dimensional model of the virtual robot. Further, information other than the three-dimensional model may be stored in the storage unit 31. For example, teaching data etc. may be stored in the storage unit 31.

The process by which information is stored in the storage unit 31 is not important. For example, information may be stored in the storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage unit 31. The storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, etc.

The marker positional relationship acquisition unit 32 acquires the relative positional relationship between the reference marker 6 existing in the real environment and the display device 4. Acquiring the relative positional relationship between the reference marker 6 and the display device 4 may, for example, be acquiring the relative positional relationship between the marker coordinate system, which is the local coordinate system of the reference marker 6, and the display coordinate system, which is the local coordinate system of the display device 4. This relative positional relationship may be indicated, for example, by a homogeneous transformation matrix indicating the transformation between the two coordinate systems. The method by which the marker positional relationship acquisition unit 32 acquires this relative positional relationship is not important. For example, the marker positional relationship acquisition unit 32 may receive an image captured by the camera of the display device 4, and may acquire a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system using three or more feature points of the reference marker 6 included in the image. The homogeneous transformation matrix may be acquired in the display device 4. In this case, the marker positional relationship acquisition unit 32 may receive a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system from the display device 4. In other words, the acquisition of the relative positional relationship by the marker positional relationship acquisition unit 32 may be the acceptance of the relative positional relationship.

The reception unit 33 receives operations on the three-dimensional model of the virtual robot. The reception unit 33 also receives an instruction to register the position and posture of the end effector of the virtual robot. The registration instruction may be received, for example, when the end effector is in a desired position and posture in the display image of the three-dimensional model of the virtual robot. The reception unit 33 also receives an instruction to move the virtual robot. The movement instruction may be input, for example, to resolve interference with peripheral devices that occurs when the end effector is in the registered position and posture at the current placement position of the virtual robot, or may be input for other reasons.

The operation on the three-dimensional model may be, for example, an operation for changing the position or posture of at least a part of the three-dimensional model. The instruction for the operation may be received, for example, through an input device such as a teaching pendant present in the real environment, or through a virtual input interface such as a virtual button or a virtual teaching pendant displayed on the display of the display device 4. The virtual input interface may be displayed on the display of the display device 4 in response to an operation such as an air tap, and when a button or the like is selected by the worker's finger or a pointing device, an input corresponding to the operation of the button or the like may be passed from the display device 4 to the reception unit 33. In addition, the position or posture of at least a part (for example, a hand tip such as an end effector) of the three-dimensional model of the virtual robot may be changed by a gesture of the worker's hand or the like. In this case, for example, when the worker performs an operation of pinching the three-dimensional model displayed on the display with his/her hand (hold operation), the part pinched by the hand is identified as the object of operation, and an operation of changing the position or posture of the identified object is performed by changing the position or posture of the hand, and the operation on the identified object may be ended by ending the pinching operation. In this case, for example, information indicating the target of the operation (e.g., information indicating a position or a part in the three-dimensional model) and information indicating the content of the operation (e.g., information indicating a change in position or a change in posture) may be passed from the display device 4 to the reception unit 33, or the hand tracking results acquired by the display device 4 may be passed to the reception unit 33, and the target of the operation and the content of the operation may be specified in the image generation device 3. When the information indicating the target of the operation and the information indicating the content of the operation are acquired by the display device 4, the display device 4 may be able to access information on the current three-dimensional model of the virtual robot in the virtual space, which is held by the image generation device 3. Note that, when an operation is performed using the worker's hand, the display device 4 may have a camera, and the position of the worker's hand on the display may be specified by performing hand tracking of the worker's hand photographed by the camera. Also, a method for converting an operation on a display image of a three-dimensional model according to a gesture of the worker's hand or the like into a change in the position or posture of the three-dimensional model in a three-dimensional virtual space is already known, and a detailed description thereof will be omitted.

The reception of instructions to register the position and orientation of the end effector of the virtual robot may be performed, for example, via an input device in the real environment, or may be performed via a virtual input interface displayed on the display of the display device 4. Good too. Further, the movement instruction of the virtual robot may be received through, for example, an input device in the real environment, a virtual input interface displayed on the display of the display device 4, or the movement instruction of the virtual robot may be received through a gesture operation. This may be done by moving the dimensional model. The movement instruction for the virtual robot accepted via the input device or the virtual input interface may be, for example, one that indicates the direction of movement or the degree of movement using numerical values. In this case, for example, a numerical value corresponding to a vector indicating the direction and degree of movement may be accepted. Movement instructions for the virtual robot that are accepted through gesture operations include, for example, an action of pinching the proximal portion of the three-dimensional model of the virtual robot with the hand, an action of changing the position of the hand, and an action of ending the pinching action. It may also include. The accepted movement instruction is not particularly limited as long as it is information that indicates how the placement position of the virtual robot will change as a result. It may be a homogeneous transformation matrix indicating the transformation of , or it may be information indicating other movement contents.

The receiving unit 33 may also receive information, instructions, etc. other than those described above. For example, the receiving unit 33 may receive information input from an input device or the display device 4, or may receive information transmitted via a wired or wireless communication line. Note that the accepting unit 33 may or may not include a device for accepting requests (for example, an input device, a communication device, etc.). Further, the reception unit 33 may be realized by hardware or by software such as a driver that drives a predetermined device.

The registration unit 34 registers the position and orientation of the end effector in the three-dimensional model of the virtual robot when the registration instruction is received by the reception unit 33. The position and orientation of the end effector may be, for example, the position and orientation of a part of the end effector. For example, when the end effector is a welding torch, the position and orientation of the end effector may be the position and orientation of the tip of the welding torch. Further, the position of the end effector may be, for example, the position of a TCP (Tool Center Point) of the virtual robot. The position and orientation of the end effector may be, for example, the position and orientation in the world coordinate system. The world coordinate system may be, for example, a marker coordinate system. For example, the registration unit 34 can acquire the position and orientation of the end effector in the robot coordinate system from the image generation unit 36. Further, the positional relationship between the robot coordinate system and the world coordinate system such as the marker coordinate system is, for example, a predetermined positional relationship or a positional relationship acquired by the robot positional relationship acquisition unit 35. Therefore, the registration unit 34 can use this information to specify the position and orientation of the end effector in the world coordinate system such as the marker coordinate system. The position of the end effector may be indicated by coordinate values in a predetermined coordinate system, for example. Further, the attitude of the end effector may be indicated by, for example, roll, pitch, yaw angle, Euler angle, or the like. Further, the information on the position and posture registered by the registration unit 34 may be any type of information as long as the position and posture of the end effector can be specified as a result. Registering the position and orientation of the end effector may mean, for example, storing information indicating the position and orientation in the storage unit. The storage unit may be, for example, the storage unit 31 or another recording medium. In this embodiment, the latter case will mainly be explained.

The robot positional relationship acquisition unit 35 acquires a new positional relationship between the reference marker 6 and the virtual robot in response to reception of the movement instruction by the reception unit 33. The new positional relationship between the reference marker 6 and the virtual robot may be, for example, information indicating the placement position of the virtual robot in the world coordinate system after the virtual robot has moved in response to a movement instruction. The positional relationship of the virtual robot before and after movement (for example, a homogeneous transformation matrix regarding the robot coordinate system of the virtual robot before and after movement) can be specified based on the movement instruction. Further, the position of the virtual robot in the robot coordinate system before movement in the world coordinate system such as the marker coordinate system is known. Therefore, using such information, the robot positional relationship acquisition unit 35 can specify the position of the virtual robot in the robot coordinate system after movement in the world coordinate system such as the marker coordinate system. That is, the robot positional relationship acquisition unit 35 can acquire a new positional relationship between the reference marker 6 and the virtual robot. This new positional relationship may be, for example, a homogeneous transformation matrix that indicates transformation between the marker coordinate system and the robot coordinate system of the virtual robot at the placement position after movement.

Based on the three-dimensional model of the virtual robot stored in the memory unit 31 and the relative positional relationship acquired by the marker positional relationship acquisition unit 32, the image generation unit 36 generates a display image for displaying the three-dimensional model of the virtual robot arranged to have a predetermined positional relationship with the reference marker 6 in response to the operation accepted by the acceptance unit 33. When a new positional relationship is acquired by the robot positional relationship acquisition unit 35, the image generation unit 36 generates a display image for displaying the three-dimensional model of the virtual robot arranged to have a new positional relationship with the reference marker 6, the three-dimensional model being the registered position and posture of the end effector.

The relative positional relationship between the reference marker 6 and the display device 4 is acquired by the marker positional relationship acquisition unit 32. Furthermore, the positional relationship between the reference marker 6 and the three-dimensional model of the virtual robot before movement, that is, the initial positional relationship, is fixed, and the positional relationship after movement according to the movement instruction, that is, the new positional relationship, is the robot position. It is acquired by the relationship acquisition unit 35. Therefore, using this information, the image generation unit 36 can specify the relative positional relationship between the three-dimensional model of the virtual robot and the display device 4. Therefore, the image generation unit 36 arranges a three-dimensional model of the virtual robot in the virtual space, and specifies the position and orientation of the display device 4 that will be the initial positional relationship or the new positional relationship with respect to the three-dimensional model. be able to. Then, the image generation unit 36 generates a two-dimensional display image for displaying the three-dimensional model by rendering the three-dimensional model of the virtual robot in the virtual space based on the position and orientation of the display device 4. be able to. Note that the angle of each joint of the virtual robot will be the initial value when no operation is performed, and will be the value after the operation when the operation is performed. Further, when generating a display image of a virtual robot in which the end effector is in a registered position and posture, the angle of each joint is in accordance with the registered position and posture of the end effector. The position and orientation of the end effector registered by the registration unit 34 are, for example, the position and orientation in a world coordinate system such as a marker coordinate system. Further, a new placement position of the virtual robot in the world coordinate system such as the marker coordinate system can be specified based on the new positional relationship acquired by the robot positional relationship acquisition unit 35. Therefore, the image generation unit 36 can specify the position and orientation of the end effector in the local coordinate system (for example, robot coordinate system) of the new placement position of the virtual robot, and obtain the angle of each joint accordingly. be able to. The angle of each joint after operation and the angle of each joint according to the registered position and posture of the end effector can be determined using, for example, the position and posture of the hand in the three-dimensional model of the virtual robot, similar to the real robot. It may also be calculated by inverse kinematics. When the virtual robot is operated, the shape of the three-dimensional model in the virtual space is changed accordingly. Furthermore, if the position or orientation of the display device 4 changes in the real environment, the position or direction of the viewpoint in the virtual space will be changed accordingly. Then, by performing rendering after the change, a display image of the three-dimensional model after the operation and a display image of the three-dimensional model after the position and orientation of the display device 4 have been changed are generated. Note that the image generation unit 36 generates a display image so that when the display image of the three-dimensional model of the virtual robot is displayed on the display of the display device 4, the size of the display image matches the real environment. shall be taken as a thing. That is, the three-dimensional model of the virtual robot displayed on the display of the display device 4 and the real robot placed in the real environment so as to have the same relative positional relationship as the three-dimensional model can be displayed via the display device 4. Display images are generated so that they have the same size when viewed from the front.

Note that when the new positional relationship is acquired by the robot positional relationship acquisition unit 35, the image generation unit 36 registers the end effector of the virtual robot arranged to have a new positional relationship with the reference marker 6. If the angle of each joint can be calculated using inverse kinematics in a certain position and posture, a display image is generated according to the new positional relationship, and if not, a display image is generated according to the new positional relationship. It is not necessary to generate an image. The angle of each joint of the virtual robot cannot be calculated by inverse kinematics, for example, if there is no solution in inverse kinematics calculations, or if there is a solution in inverse kinematics calculations, but the solution is not possible. At least one of these may exceed the preset range of motion of each joint. The operating range of each joint may be provided, for example, to prevent interference with the virtual robot's casing, etc., or may be provided to prevent cable breakage, avoidance of singular points, and the like.

Furthermore, it may not be possible to operate the virtual robot according to the accepted operation. For example, an operation to move the hand of the virtual robot beyond its movable range or to rotate it beyond its rotatable range may be accepted. In such a case, the image generation unit 36 does not need to generate a display image according to the operation, or may display a three-dimensional model such as a virtual robot that has moved or rotated within the possible range. An image may also be generated.

The virtual robot is displayed so that it has a predetermined positional relationship (for example, an initial positional relationship or an acquired positional relationship) with the reference marker 6 placed in the real environment. Therefore, even if the orientation of the display device 4 is changed, the display position of the three-dimensional model of the virtual robot in the real environment will not change. In addition, the display image generated by the image generation unit 36 may be, for example, an opaque image that does not allow the image behind it to be seen, or a translucent image that allows the image behind it to be seen.

The output unit 37 outputs the display image generated by the image generation unit 36 to the display device 4. When outputting the display image, the output unit 37 may output only the display image. In this case, the display image is displayed on the display device 4 by being superimposed on an image of the real environment or the real environment itself. On the other hand, if the display device 4 has a non-transparent display and an image of the real environment captured by the display device 4 is accepted by the image generation device 3, the result of combining the image of the real environment and the display image may be output to the display device 4. This combination may be performed, for example, by a combination unit (not shown) included in the image generation device 3.

When the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6, for example, as shown in FIG. 1, the three-dimensional model 10 of the virtual robot may be displayed next to the reference marker 6. The reference marker 6 and the three-dimensional model 10 of the virtual robot in FIG. 1 are schematic representations of the situation as viewed by the operator operating the virtual robot via the display device 4.

Furthermore, regarding displaying a 3D model of a virtual robot and operating the 3D model of the virtual robot using a virtual input interface of a display device, operator's gestures, etc., see, for example, the following document 1. As shown in 3. to 3, it is already known, and detailed explanation thereof will be omitted.
Document 1: JP2017-100234A Document 2: JP2020-055075A Document 3: JP2020-069538A

In addition, when the end effector of the virtual robot arranged to have a new positional relationship with the reference marker 6 cannot calculate the angles of each joint by inverse kinematics in a state where the end effector is in the registered position and posture, the output unit 37 may output a message indicating that the virtual robot cannot move to the new positional relationship. The message indicating that the virtual robot cannot move to the new positional relationship may be, for example, a message indicating that the virtual robot cannot move to a new arrangement position corresponding to the new positional relationship. This output may be, for example, an image or sound output to the display device 4, a display on another display device, a transmission to a specified device via a communication line, a printing by a printer, a sound output by a speaker, storage on a recording medium, or a transfer to another component. The output unit 37 may or may not include a device that performs output (for example, a display device, a speaker, etc.). The output unit 37 may be realized by hardware, or may be realized by software such as a driver that drives those devices.

Next, the operation of the image generating device 3 will be described with reference to the flowchart of FIG.
(Step S101) The image generation unit 36 generates a display image of a three-dimensional model of the virtual robot, and the output unit 37 outputs the generated display image to the display device 4. The reception unit 33 may also receive an operation on an end effector of the displayed virtual robot. When an operation is received, a display image of a three-dimensional model of the virtual robot in which the position or posture of the end effector has been changed in response to the operation may be generated and output. Details of this process will be described later using the flowchart in FIG. 3.

(Step S102) The reception unit 33 determines whether a registration instruction has been received. If a registration instruction has been received, the process proceeds to step S103; if not, the process returns to step S101.

(Step S103) The registration unit 34 registers the position and orientation of the end effector in the three-dimensional model of the virtual robot at that time.

(Step S104) The reception unit 33 determines whether a movement instruction for the virtual robot has been received. If a movement instruction is accepted, the process proceeds to step S105, and if not, the process of step S104 is repeated until a movement instruction is received.

(Step S105) In response to receiving the movement instruction, the robot positional relationship acquisition unit 35 acquires the new positional relationship between the reference marker 6 and the virtual robot after the movement.

(Step S106) The image generation unit 36 determines whether to generate a display image according to the new positional relationship. For example, if the image generation unit 36 can calculate the angles of each joint by inverse kinematics when the end effector of the virtual robot arranged to have a new positional relationship with the reference marker 6 is in the registered position and posture, the image generation unit 36 may determine to generate a display image according to the new positional relationship and proceed to step S107, or if not, may determine not to generate a display image according to the new positional relationship and return to step S104. When returning to step S104, the output unit 37 may output, for example, a message indicating that the virtual robot cannot be moved to the new positional relationship.

(Step S107) The image generation unit 36 generates a virtual robot arranged so as to have a new positional relationship with the reference marker 6 obtained in step S105, and a virtual robot whose end effector is in the registered position and orientation. A display image for displaying a three-dimensional model of the robot is generated. Note that the details of this process will be described later using the flowchart of FIG.

(Step S108) The reception unit 33 determines whether an instruction to end the series of processes has been received. If the instruction to end is accepted, the process returns to step S101; otherwise, the process proceeds to step S109.

(Step S109) The reception unit 33 determines whether or not a movement instruction for the virtual robot has been received. If a movement instruction has been received, the process returns to step S105; if not, the process returns to step S107.

Note that while the flowchart in FIG. 2 shows a case where the position and orientation of the end effector can only be registered at the initial position of the virtual robot, it is also possible to register the position and orientation of the end effector at the placement position after the virtual robot is moved. Also, the order of the processing in the flowchart in FIG. 2 is an example, and the order of each step may be changed as long as the same results are obtained. Also, in the flowchart in FIG. 2, the processing ends when the power is turned off or an interrupt is issued to end the processing.

Next, the process of generating a display image of a virtual robot (steps S101 and S107) in the flowchart of FIG. 2 will be described using the flowchart of FIG. 3. Note that in step S101, the object to be manipulated and displayed is the three-dimensional model of the virtual robot arranged to have an initial positional relationship with the reference marker 6, and in step S107, the object to be manipulated and displayed is 3 is a three-dimensional model of a virtual robot arranged to have a new positional relationship with the reference marker 6, but in FIG. 3, the two will be explained without making any particular distinction.

(Step S201) The reception unit 33 determines whether an operation on the three-dimensional model has been received. If an operation has been received, the process proceeds to step S203; if not, the process proceeds to step S202.

(Step S202) The image generation unit 36 determines whether to generate a display image. Then, if a display image is to be generated, the process advances to step S203, and if not, the process returns to the flowchart of FIG. 2. Note that the image generation unit 36 may, for example, periodically determine to generate a display image. By making this determination, for example, even if no operation is performed, if the position or orientation of the display device 4 is changed, a display image corresponding to the changed position or orientation will be displayed on the display device 4. will be displayed.

(Step S203) The marker positional relationship acquisition unit 32 acquires the relative positional relationship between the reference marker 6 and the display device 4.

(Step S204) The image generation unit 36 uses the three-dimensional model of the virtual robot stored in the storage unit 31 and the relative positional relationship acquired in step S203 to create a display for displaying the three-dimensional model. Generate an image. Further, if the operation is accepted, a display image for displaying the three-dimensional model changed according to the operation will be generated. Furthermore, when generating a display image in step S107, a display image for displaying a three-dimensional model of the virtual robot in which the end effector is registered in the position and orientation is generated. Further, when the position or orientation of the display device 4 is changed, a display image of a three-dimensional model is generated according to the changed position or orientation of the display device 4. It is assumed that the display image for displaying the three-dimensional model of the virtual robot is generated so as to have an initial positional relationship or a new positional relationship with the reference marker 6.

(Step S205) The output unit 37 outputs the generated display image to the display device 4. Then, the process returns to the flowchart of FIG. 2. In response to this output, for example, a three-dimensional model of a virtual robot is displayed.

Next, the operation of the image generating device 3 according to this embodiment will be described using a specific example. First, the worker places the sheet 6a on which the reference marker 6 is displayed at the position where the real robot to be introduced is to be placed. Next, the worker wears the head-mounted display device 4 on his head and starts the processing of the image generating device 3. Then, the image generating device 3 generates a display image of the virtual robot and outputs it to the display device 4 (step S101). Specifically, the image generating unit 36 determines that a display image is to be generated, and transmits an instruction to the marker positional relationship acquisition unit 32 to acquire the relative positional relationship between the reference marker 6 and the display device 4 (step S202). In response to the instruction, the marker positional relationship acquisition unit 32 acquires the relative positional relationship and transmits it to the image generating unit 36 (step S203). Upon receiving the relative positional relationship, the image generating unit 36 uses the relative positional relationship and the three-dimensional model of the virtual robot stored in the storage unit 31 to generate a display image for displaying the three-dimensional model at a predetermined initial position and initial posture, and transmits it to the output unit 37 (step S204). The output unit 37 outputs the received display image to the display device 4 (step S205). As a result, the worker can see the virtual robot superimposed on the real space.

Thereafter, the operator uses an input device such as a teaching pendant or gestures to move the tip of the welding torch, which is the displayed end effector, to the desired position and orientation, as shown in FIG. (Steps S201, S203 to S205). Note that in FIG. 4, displays of workpieces, peripheral devices, etc. placed in the real environment other than the reference marker 6 and the three-dimensional model 10 of the virtual robot are omitted. The same applies to FIG. 5, which will be described later. At this point, when the operator inputs a registration instruction via an input device or a virtual input interface, the input is accepted by the reception unit 33 and passed to the registration unit 34 (step S102). Then, the registration unit 34 acquires the position and orientation of the end effector at that time and stores it in a recording medium (not shown) (step S103).

Next, the worker inputs a movement instruction for the displayed virtual robot using an input device, a virtual input interface, or a gesture operation. This movement instruction may be input, for example, to resolve interference when the 3D model 10 of the virtual robot with the end effector in a desired position and posture interferes with a peripheral device or the like. The movement instruction is received by the reception unit 33 and passed to the robot positional relationship acquisition unit 35 (step S104). Upon receiving the movement instruction, the robot positional relationship acquisition unit 35 acquires a new positional relationship between the reference marker 6 and the virtual robot, and passes it to the image generation unit 36 (step S105).

When the new positional relationship between the reference marker 6 and the virtual robot is received, the image generation unit 36 acquires the registered position and orientation of the end effector from the registration unit 34, and determines whether the angles of each joint of the virtual robot can be calculated by inverse kinematics in a state in which the end effector of the virtual robot arranged so as to have the new positional relationship with the reference marker 6 is in the registered position and orientation (step S106). In this case, it is assumed that the angles of each joint can be calculated by inverse kinematics. Then, the image generation unit 36 generates a display image for displaying a three-dimensional model of the virtual robot arranged so as to have the new positional relationship with the reference marker 6, in which the position and orientation of the end effector are the registered position and orientation, and the output unit 37 outputs the display image to the display device 4 (steps S107, S202 to S205). As a result, as shown in FIG. 5, the operator can see a display image of the 3D model 10 of the virtual robot placed in a position after movement, with the 3D model 10a of the end effector in the registered position and orientation, and can check, for example, in the real environment, the situation when the end effector of the moved virtual robot is in the desired position and orientation. It can also be checked, for example, whether the virtual robot will interfere with peripheral devices placed in the real environment.

In addition, when the worker inputs a movement command, if, for example, the virtual robot has been moved too far and the angles of each joint of the virtual robot cannot be calculated using inverse kinematics when the end effector of the virtual robot placed in the new position is in the registered position and orientation (step S106), the input of the movement command will be accepted again. At that time, an output may be output indicating that movement to the new position is not possible, to inform the worker that a movement command that exceeds the possible range has been input.

Note that the display of the three-dimensional model 10 of the virtual robot is changed by the operator operating the displayed three-dimensional model 10 of the virtual robot (steps S107, S201, S203 to S205). Therefore, the worker can also confirm the operation of the virtual robot at the new placement position. Further, when the worker inputs a new movement instruction, the movement instruction is accepted by the reception unit 33 (step S109), and the new positional relationship is acquired and the virtual robot is displayed according to the new positional relationship. etc. are performed (steps S105 to S107). In this case as well, a 3D model of the virtual robot placed at a new position, where the position and orientation of the end effector are the registered position and orientation, is displayed. A display image will be generated and displayed on the display device 4.

As described above, the image generation device 3 according to the present embodiment generates and outputs a display image of a three-dimensional model of a virtual robot whose placement position is changed while the position and posture of the end effector are fixed. You will be able to do this. Therefore, for example, by changing the placement position of the virtual robot while the position and orientation of the welding torch of the virtual robot, which is a welding robot, is fixed at the welding start position, interference may occur with peripheral devices placed in the real environment. This makes it possible to consider the placement position of robots that are not used. Note that since the display image of the three-dimensional model 10 of the virtual robot can be displayed superimposed on an image of the real environment or the real environment itself, highly accurate placement studies can be performed. Furthermore, since there is no need to prepare three-dimensional models of workpieces in the actual environment, peripheral devices, etc., placement can be considered at low cost. In addition, depending on whether the angle of each joint can be calculated by inverse kinematics with the end effector of the virtual robot placed in a new positional relationship with the reference marker 6 in the registered position and orientation, the new It is possible to determine whether to generate a display image according to the positional relationship. For example, if the display image of the virtual robot is not displayed at a new placement position, the operator cannot move the virtual robot to that placement position. It is possible to know that the registered position and posture cannot be achieved. In addition, if the angle of each joint cannot be calculated using inverse kinematics, it is possible to notify the operator that a movement instruction that exceeds the movable range is input by outputting a message that the movement cannot be moved to a new position. It is possible to prompt the user to input the movement instruction again.

In the present embodiment, the case where one position and posture of the end effector is registered has been mainly described, but this is not necessary. Two or more positions and postures of the end effector may be registered by the registration unit 34. For example, when the end effector is a welding torch, the position and posture of the welding torch at the start position of welding and the position and posture of the welding torch at the end position of welding may be registered. In this way, when two or more positions and postures of the end effector are registered by the registration unit 34, the image generation unit 36 may generate display images for displaying two or more three-dimensional models of the virtual robot that are arranged to have a new positional relationship with the reference marker 6 when a new positional relationship is acquired by the robot positional relationship acquisition unit 35, and that are two or more positions and postures of the end effector that are registered.

More specifically, when the first position and orientation and the second position and orientation of the end effector are registered, the image generation unit 36 may generate a display image for displaying a three-dimensional model of the first virtual robot in the first position and orientation in which the end effector is registered, arranged so as to have a new positional relationship with the reference marker 6, and a three-dimensional model of the second virtual robot in the second position and orientation in which the end effector is registered, arranged so as to have a new positional relationship with the reference marker 6. In addition, if the image generation unit 36 can calculate the angle of each joint by inverse kinematics when the end effector of the first virtual robot arranged so as to have a new positional relationship with the reference marker 6 is in the registered first position and orientation, and can calculate the angle of each joint by inverse kinematics when the end effector of the second virtual robot arranged so as to have a new positional relationship with the reference marker 6 is in the registered second position and orientation, the image generation unit 36 may generate a display image according to the new positional relationship, and may not generate a display image according to the new positional relationship otherwise.

In this way, when N positions and postures of the end effector are registered, the image generation unit 36 may generate display images for displaying three-dimensional models of N virtual robots. N is an integer of 2 or more. Note that the three-dimensional models of the N virtual robots are three-dimensional models of virtual robots that have the same proximal position, that is, the same arrangement position, but different positions and postures of the hands. The image generation unit 36 also calculates the angles of each joint in a state where the end effector of the virtual robot, which is arranged to have a new positional relationship with the reference marker 6, is in the registered first to Nth positions and postures. If each can be calculated by inverse kinematics, display images of N virtual robots are generated according to the new positional relationships; otherwise, display images are generated according to the new positional relationships. You don't have to.

Processing such as registration of two positions and postures of the end effector may be performed as follows, for example. First, the operator registers the position and orientation of the first end effector. When this registration is completed, the 3D model of the virtual robot with the position and orientation of the first end effector registered is displayed, and the 3D model of the new virtual robot is displayed at the same placement position. Ru. Then, the worker operates the three-dimensional model of the new virtual robot to register the position and orientation of the second end effector. Thereafter, when the operator moves the placement positions of the three-dimensional models of the two virtual robots by a movement instruction, the three-dimensional models of the two virtual robots are displayed at the new placement positions accordingly. Note that the positions and postures of the hands of the three-dimensional model of each virtual robot are the registered positions and postures. Here, it is assumed that the angle of each joint can be calculated using inverse kinematics at the new arrangement position.

In addition, when two or more positions and postures of the end effector are registered, it may not be easy to find a placement position of the virtual robot where the end effector can be in the two or more registered positions and postures. Therefore, for example, when two or more positions and postures of the end effector are registered by the registration unit 34, the robot positional relationship acquisition unit 35 may acquire multiple new positional relationships set at a predetermined interval. Then, the image generation unit 36 may specify a new positional relationship among the multiple new positional relationships, in which the angle of each joint can be calculated by inverse kinematics in each state in which the end effector of the virtual robot is in the two or more registered positions and postures. In addition, this new positional relationship is the positional relationship between the reference marker 6 and the virtual robot. After that, the image generation unit 36 may generate a display image for displaying each of the three-dimensional models of the virtual robot that are placed so as to be in the specified positional relationship with the reference marker 6 and that are in the two or more registered positions and postures of the end effector. In this way, a placement position of a virtual robot where the end effector can be in two or more registered positions and postures may be automatically found, and display images of the multiple virtual robots at the placement positions may be generated. In this case, the worker can easily know the placement position of a virtual robot where the end effector can be in two or more registered positions and postures.

More specifically, the robot positional relationship acquisition unit 35 may, for example, set multiple new placement positions at predetermined intervals around the current placement position of the virtual robot, and acquire new positional relationships with the reference marker 6 for each of the multiple new placement positions. For example, as shown in FIG. 6, when the current placement position 30 is the position of the reference marker 6, the robot positional relationship acquisition unit 35 may set multiple lattice points arranged at a predetermined interval with the current placement position 30 as the center as new placement positions 31. As an example, the multiple lattice points may be arranged on the surface on which the reference marker 6 is arranged. In FIG. 6, all black circles other than the current placement position 30 are new placement positions 31. For example, the new placement position 31 may be set in the robot coordinate system at that time. Since the relationship between the robot coordinate system and the marker coordinate system is known, the robot positional relationship acquisition unit 35 can acquire multiple new positional relationships with the reference marker 6 for each of the multiple new placement positions 31. Note that each of the placement positions 30 and 31 may be, for example, a position where the center point of the end face on the base end side of the three-dimensional model of the virtual robot is located when the virtual robot is placed. Also, the orientation of the virtual robot when the three-dimensional model of the virtual robot is placed at each of the placement positions 30 and 31 may be determined, for example, according to the orientation of the reference marker 6, or may be determined according to another criterion.

The image generation unit 36 selects one new positional relationship from among the acquired new positional relationships, and the end of the virtual robot arranged so as to have the reference marker 6 and the selected new positional relationship. When the angle of each joint can be calculated by inverse kinematics in each state where the effector is in two or more registered positions and postures, the selected new positional relationship is used as the position for generating the display image. It may also be specified as a relationship. The image generation unit 36 generates a three-dimensional model of the virtual robot arranged in a specified positional relationship with the reference marker 6, and has two or more positions and orientations in which the end effector is registered. A display image for displaying a three-dimensional model of a virtual robot may be generated. On the other hand, the angle of each joint is calculated by inverse kinematics in each state in which the end effector of the virtual robot, which is placed in the selected new positional relationship with the reference marker 6, is in two or more registered positions and postures. If this is not possible, the image generation unit 36 may select a new positional relationship that has not been selected so far, and repeat the same process. Note that the selection of a new positional relationship may be performed, for example, in a predetermined order or at random. For example, when a new positional relationship is found that allows the angle of each joint to be calculated by inverse kinematics for two or more registered positions and postures of the end effector, the image generation unit 36 generates the new positional relationship. It is possible to generate a display image of a virtual robot arranged according to the positional relationship, and to calculate the angle of each joint using inverse kinematics for two or more registered positions and postures of the end effector. After finding a plurality of new positional relationships, one new positional relationship is identified among the new positional relationships, and a display image of the virtual robot arranged according to the new positional relationship is generated. It's okay. In the latter case, for example, one new positional relationship may be randomly specified from a plurality of new positional relationships, and one of the plurality of placement positions corresponding to the plurality of new positional relationships is A new positional relationship corresponding to the placement position closest to the center of gravity of the placement positions may be specified, and the placement position furthest from the tip position of the end effector among the plurality of placement positions corresponding to the plurality of new positional relationships. A new positional relationship corresponding to the above may be specified, or other new positional relationships may be specified.

Note that, as described above, because the motion range of each joint of the virtual robot is set, the first axis of the virtual robot, that is, the joint on the proximal end side, may not be able to rotate 360 degrees. For example, the operating range of the first axis may be limited to a range of -160 degrees to 160 degrees, that is, a total range of 320 degrees. In such a case, the image generation unit 36 calculates the inverse kinematics at K angles obtained by rotating the virtual robot by (360/K) degrees at each placement position of the virtual robot, and When the angle of each joint can be calculated using the method described above, it may be possible to calculate the angle of each joint at the arrangement position using inverse kinematics. Note that K is an integer of 1 or more calculated by the following formula. In the following equation, ceil is a ceiling function, and M is the angle (deg) of the operating range of the first axis of the virtual robot.
K = ceil (360/M)

For example, when M = 320 degrees as described above, K = 2, and the image generation unit 36 may place the virtual robot at an arbitrary angle in a certain placement position and perform inverse kinematics calculations. If the calculations fail to calculate the angles of each joint, the image generation unit 36 may rotate the virtual robot 180 degrees (= 360/K) at that placement position and perform inverse kinematics calculations. Then, if the calculations fail to calculate the angles of each joint, the angles of each joint may be calculated by inverse kinematics at that placement position. Note that the virtual robot rotates about an axis normal to the placement surface of the virtual robot. If K = 1, it is not necessary to rotate the virtual robot at each placement position.

In addition, when performing inverse kinematics calculations for two or more registered positions and orientations of the end effector, the image generation unit 36 allows the end effector to continuously move between the two or more registered positions and orientations of the end effector. Even when the end effector is unable to move, it may be determined that the angle of each joint cannot be calculated by inverse kinematics in each state where the end effector is in two or more registered positions and postures. The fact that the end effector cannot move continuously between two or more registered positions and postures means, for example, that the end effector cannot move continuously between two or more registered positions and postures. When moving to a posture, the angle of the joint, which is in the range of motion from -180 degrees to 180 degrees, may change to -179 degrees, -180 degrees, and -181 degrees. This is because the joint angle from -180 degrees to -181 degrees means that the joints rotate 359 degrees in a real robot, and such a movement is not possible.

Further, in this embodiment, the case where the virtual robot after movement is displayed in response to the acceptance of a movement instruction has been mainly described, but the virtual robot may also be displayed in real time even during movement. good. In this case, the image generation unit 36 may generate a display image of the moving virtual robot in real time even during the movement, for example, when a movement instruction for the virtual robot is input by a gesture operation. That is, the processes of steps S104 to S107 may be repeated from the start of movement of the virtual robot to the end of movement. Note that in this case, only the processes of steps S203 to S205 may be performed in step S107 until the movement is completed. By doing so, the worker can check the posture of the virtual robot even during movement.

In addition, in this embodiment, the case where the position and posture of the end effector of the virtual robot are registered has been mainly described, but only the position of the end effector of the virtual robot may be registered. In this case, the position and posture of the end effector in the above description may be the position of the end effector. More specifically, the reception unit 33 may receive an instruction to register the position of the end effector of the virtual robot, the registration unit 34 may register the position of the end effector in the three-dimensional model of the virtual robot, and the image generation unit 36 may generate a display image for displaying the three-dimensional model of the virtual robot that is arranged so as to have a new positional relationship with the reference marker 6 and is the position where the end effector is registered. The posture of the end effector in the display image may be any posture. In this way, when only the position of the end effector is registered, the posture of the end effector is any posture when the end effector is in the registered position. In this case, if the angle of each joint can be calculated by inverse kinematics when the end effector of the virtual robot, which is arranged to have a new positional relationship with the reference marker 6, is in the registered position, a display image according to the new positional relationship is generated, and if not, a display image according to the new positional relationship does not have to be generated. The state in which the end effector of the virtual robot is in the registered position may be a state in which the end effector of the virtual robot is in the registered position and in any posture.

The registration of the position and orientation of the end effector may also be the registration of the position and orientation of a location that has a predetermined positional relationship with the end effector. However, it is assumed that the position and orientation of that location and the position and orientation of the end effector have a one-to-one relationship. In other words, once the position and orientation of that location are determined, the position and orientation of the end effector are also uniquely determined. In this case, registering the position and orientation of that location is substantially the same as registering the position and orientation of the end effector. Therefore, it may be considered that the registration of the position and orientation of the end effector includes the registration of such a location. Furthermore, the position and orientation of that location may be used to generate a display image, etc. The location may be, for example, a location that is a predetermined distance away from the end effector in a predetermined direction. As an example, the location may be a location on a virtual robot. The same applies when only the position of the end effector is registered.

Further, in this embodiment, the case where the virtual robot is placed on the floor, that is, the case where it is placed on the floor, has been mainly described, but the placement position of the virtual robot does not matter. For example, the virtual robot may be a wall-mounted virtual robot placed on a wall, or a ceiling-suspended virtual robot placed on the ceiling. In these cases, the placement surface of the virtual robot may be changed to a wall or ceiling other than the floor surface by a movement instruction.

In addition, in the above embodiment, each process or function may be realized by centralized processing by a single device or a single system, or may be realized by distributed processing by multiple devices or multiple systems. For example, at least a portion of the configuration of image generating device 3 may be physically included in a device having a display. Therefore, the division of devices shown in FIG. 1 may be considered to be for convenience according to function, rather than according to physical devices.

Furthermore, in the above embodiments, each process or each function may be realized by being centrally processed by a single device or a single system, or may be realized by distributed processing by multiple devices or multiple systems. This may be realized by

Further, in the above embodiment, when two or more components included in the image generation device 3 have a communication device, an input device, etc., even if the two or more components physically have a single device. or may have separate devices.

Furthermore, in the embodiments described above, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of execution, the program execution section may execute the program while accessing the storage section or recording medium. Further, the program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). good. Further, this program may be used as a program constituting a program product. Further, the number of computers that execute the program may be one or more. That is, centralized processing or distributed processing may be performed.

The above embodiments are merely examples for specifically implementing the present invention, and are not intended to limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not by the description of the embodiments, and is intended to include modifications within the literal scope of the claims and within the scope of equivalent meanings.

3 image generation device, 4 display device, 6 reference marker, 31 storage unit, 32 marker positional relationship acquisition unit, 33 reception unit, 34 registration unit, 35 robot positional relationship acquisition unit, 36 image generation unit, 37 output unit

Claims (5)

a storage unit storing a three-dimensional model of a virtual robot corresponding to the real robot;
a marker positional relationship acquisition unit that acquires the relative positional relationship between a reference marker existing in the real environment and a display device that displays the image superimposed on the image of the real environment or the real environment itself;
a reception unit that receives an operation on a three-dimensional model of the virtual robot, an instruction to register the position or position and orientation of an end effector of the virtual robot, and an instruction to move the virtual robot;
Based on the three-dimensional model of the virtual robot and the relative positional relationship, the three-dimensional model of the virtual robot is arranged in a predetermined positional relationship with the reference marker in accordance with the received operation. an image generation unit that generates a display image for display;
an output unit that outputs the display image to the display device;
a registration unit that registers the position or position and orientation of the end effector in the three-dimensional model of the virtual robot when the registration instruction is accepted;
a robot positional relationship acquisition unit that acquires a new positional relationship between the reference marker and the virtual robot in response to receiving a movement instruction;
The image generation unit is a three-dimensional model of a virtual robot arranged so as to have a new positional relationship with the reference marker when a new positional relationship is acquired by the robot positional relationship acquisition unit, An image generation device that generates a display image for displaying a three-dimensional model of a virtual robot in which an effector is registered in a position or in a position and orientation. The image generation unit may generate a registered position of an end effector of a virtual robot arranged to have a new positional relationship with the reference marker when a new positional relationship is acquired by the robot positional relationship acquisition unit; If the angle of each joint can be calculated by inverse kinematics in a certain position and posture, a display image is generated according to the new positional relationship, and if not, a display image is generated according to the new positional relationship. The image generation device according to claim 1, wherein the image generation device does not generate a displayed image. The image generating device according to claim 2, wherein the output unit outputs a message indicating that the end effector of the virtual robot, arranged to have a new positional relationship with the reference marker, cannot move to the new positional relationship if the angles of each joint cannot be calculated by inverse kinematics when the end effector is in a registered position or position and posture. The image generation unit is configured to generate a new position between the reference marker and the new position when two or more positions or positions and orientations of the end effector are registered by the registration unit and a new positional relationship is acquired by the robot positional relationship acquisition unit. A display image for displaying two or more three-dimensional models of virtual robots arranged in a relationship, the three-dimensional models of two or more virtual robots having two or more registered positions or positions and postures of end effectors. The image generation device according to claim 1, which generates an image. the robot positional relationship acquisition unit acquires a plurality of new positional relationships set at a predetermined interval when two or more positions or positions and orientations of the end effector are registered by the registration unit;
4. The image generating device according to claim 2 or 3, wherein the image generation unit identifies a new positional relationship among the plurality of new positional relationships in which the angle of each joint can be calculated by inverse kinematics in each state in which the end effector of the virtual robot is at two or more registered positions or positions and postures, and generates a display image for displaying a three-dimensional model of the virtual robot arranged to be in the identified positional relationship with the reference marker, the three-dimensional model of the two or more virtual robots being at two or more registered positions or positions and postures in which the end effector is at.

Images