JP2023017437A - Image processing device and robot arrangement method - Google Patents

Abstract

To solve the problem that it is difficult to grasp a situation of an arranged robot from information about a size or the like prescribed in a catalog.SOLUTION: An image processing device 3 comprises: a storage part 31 that stores a three-dimensional model of a virtual robot; a receiving part 32 that receives operation to the three-dimensional model of the virtual robot; a positional relation obtaining part 33 that obtains a relative positional relation between a reference marker existing in an actual environment and a display device 4 that displays an image while superimposing the image on an image of the actual environment or on the actual environment; an image generating part 34 that generates a display image for displaying the operated three-dimensional model so that the model is in a predetermined positional relation with the reference marker, on the basis of the relative positional relation with the three-dimensional model; and an output part 35 that outputs the display image to the display device 4. This enables the actual robot to be arranged after confirming a situation of the arranged actual robot, using the virtual robot.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus for displaying virtual robots and a robot placement method for placing real robots.

Conventionally, when introducing a new robot, information such as the size and reachable range of the robot to be introduced was checked in a catalog, and the robot to be introduced was selected and the placement position was decided.

However, from the information in the catalog, it is not easy to know the reachable range of the robot, whether the robot can take the desired posture, and whether it can avoid obstacles in the environment where the robot is placed. There was a problem. Another problem is that it is difficult to actually arrange the robots when selecting the robots. Furthermore, it is also possible to convert the space in which the robots are to be arranged into three-dimensional data and examine the arrangement during simulation.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. It is an object of the present invention to provide a robot placement method.

To achieve the above object, an image processing apparatus according to an aspect of the present invention includes a storage unit that stores a three-dimensional model of a virtual robot, a reception unit that receives operations on the three-dimensional model of the virtual robot, a positional relationship acquiring unit that acquires a relative positional relationship between a reference marker that is used as a reference marker and a display device that displays an image superimposed on an image of the real environment or the real environment itself; an image generation unit for generating a display image for displaying the manipulated three-dimensional model so as to have a predetermined positional relationship with the reference marker; and an output unit for outputting the display image to a display device. It is.

According to the image processing apparatus and the like according to one aspect of the present invention, the situation when the robot is arranged, for example, the reachable range of the robot, whether the robot can take a desired posture, whether the robot can take an existing obstacle, It is possible to easily check in advance using a virtual robot whether or not it is possible to avoid Therefore, according to the confirmation result, for example, it is possible to select a robot suitable for the installation location or arrange the actual robot at an appropriate position.

Schematic diagram showing the configuration of a robot control system according to an embodiment of the present invention. 3 is a flow chart showing the operation of the image processing apparatus according to the same embodiment; 3 is a flow chart showing a robot placement method according to the same embodiment; FIG. 11 is a diagram showing an example of reference markers arranged in the same embodiment; A diagram showing an example of a virtual robot displayed so as to have a predetermined positional relationship with reference markers in the same embodiment. A diagram showing an example of a real robot arranged in the same embodiment.

An image processing apparatus and a robot arrangement method according to the present invention will be described below using embodiments. In the following embodiments, constituent elements and steps with the same reference numerals are the same or correspond to each other, and repetitive description may be omitted. The image processing apparatus according to this embodiment displays a three-dimensional model of a virtual robot so as to have a predetermined positional relationship with reference markers.

FIG. 1 is a block diagram showing the configuration of a robot control system 100 according to this embodiment. A robot control system 100 according to this embodiment is for operating a virtual robot, and includes an image processing device 3, a display device 4, and a sheet 6a on which reference markers 6 are displayed. Note that the image processing device 3 and the display device 4 may be connected by wire or wirelessly, for example.

The virtual robot to be operated is configured by a three-dimensional model existing in a virtual environment and corresponds to a real robot. That is, the virtual robot is the same as the real robot, except that it is configured as a three-dimensional model. For example, it may have the same size and configuration as the real robot. , etc. may be changeable.

Real robots are typically industrial robots and may be manipulators with multiple arms (links) connected by motor-driven joints. The real robot may be, for example, a vertical articulated robot or a horizontal articulated robot. Further, the actual robot may be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. good too. A real robot is a robot that exists in a real environment. The real environment is the environment of the real space.

The image processing device 3 generates a display image for displaying a three-dimensional model of the virtual robot so as to have a predetermined positional relationship with the reference marker 6 and outputs the display image to the display device 4 . Details of the image processing device 3 will be described later.

The display device 4 displays the image superimposed on the image of the real environment or the real environment itself. In other words, the operator who operates the virtual robot can see both the images of the real environment and the virtual environment on the display device 4 . The display device 4 may be a wearable display device worn on the head by the operator who operates the virtual robot, or may be a display device that is a portable information processing terminal such as a tablet terminal. . The wearable display device may be, for example, a head-mounted display. Moreover, the display device 4 may have, for example, a transmissive display. In this case, the display device 4 displays the image superimposed on the actual environment itself. HoloLens (registered trademark), for example, is known as a wearable display device 4 having a transmissive display. The display device 4 having such a transmissive display can also be considered as a display device for realizing mixed reality (MR). The display device 4 may also have, for example, a non-transmissive display. In this case, the display device 4 displays the image superimposed on the image of the real environment. Therefore, the display device 4 having a non-transmissive display preferably has a camera for photographing the real environment or is connected to a camera for photographing the real environment. Images of the real environment captured by the camera are displayed in real time on the non-transmissive display. Oculus Quest, for example, is known as a wearable display device 4 having a non-transmissive display. The display device 4 having such a non-transmissive display can also be considered as a display device for realizing augmented reality (AR). The display device 4, which is a portable information processing terminal such as a tablet terminal, has, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. In this embodiment, the case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

The reference marker 6 is a predetermined two-dimensional image. The reference marker 6 may be, for example, an AR marker, a QR code (registered trademark), or any other two-dimensional image having a predetermined shape. The size of the reference marker 6 may be predetermined, for example. In this embodiment, the case where the reference marker 6 is displayed on the sheet 6a will be mainly described. The reference marker 6 may be printed on a sheet 6a made of paper or resin, for example. Also, the sheet 6a may display, for example, one or more figures 6b used when arranging the real robot. This figure 6b is used for positioning the real robot. For example, it may be a figure showing the positions of screw holes for fixing the real robot. It may be a figure indicating a position, or another figure used for positioning when arranging the real robot. The figure 6b is not particularly limited as long as it is a figure used for positioning. ) or any other figure whose position can be specified. By arranging the real robot using this figure 6b, it is possible to prevent the position of the displayed virtual robot from deviating from the arrangement position of the real robot. The graphic 6b is used to arrange the real robot at the position of the virtual robot displayed so as to have a predetermined positional relationship with respect to the reference marker 6. FIG. Therefore, it is assumed that the graphic 6b is displayed in a predetermined positional relationship with the reference marker 6 so that the real robot can be arranged in that manner.

As shown in FIG. 1, the image processing apparatus 3 according to this embodiment includes a storage unit 31, a reception unit 32, a positional relationship acquisition unit 33, an image generation unit 34, and an output unit .

The storage unit 31 stores a three-dimensional model of the virtual robot. This three-dimensional model of the virtual robot may be, for example, a three-dimensional model of a virtual robot corresponding to the real robot whose introduction is being considered. The storage unit 31 may store three-dimensional models of a plurality of virtual robots. The three-dimensional models of multiple virtual robots correspond to multiple real robots, respectively. For example, when there are a plurality of real robots that are introduction candidates, the storage unit 31 may store three-dimensional models of a plurality of virtual robots corresponding to the plurality of real robots. Information other than the three-dimensional model may also be stored in the storage unit 31 . For example, teaching data or the like may be stored in the storage unit 31 . The process by which information is stored in the storage unit 31 does not matter. For example, information may be stored in the storage section 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage section 31 . The storage unit 31 is preferably implemented by a non-volatile recording medium, but may be implemented by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

The reception unit 32 receives an operation on the three-dimensional model of the virtual robot. The operation on the 3D model may be, for example, an operation for changing the position or orientation of at least part of the 3D model. Instructions for operation and teaching may be accepted, for example, via an input device such as a teaching pendant that exists in the real environment, and a virtual input such as a virtual button or a virtual teaching pendant displayed on the display of the display device 4 may be used. It may be done through an interface. The virtual input interface is displayed on the display of the display device 4 according to an operation such as air tapping, and when a button or the like is selected by the operator's finger or pointing device, an input corresponding to the operation of the button or the like is performed. It may be passed from the display device 4 to the reception unit 32 . In addition, the position and posture of at least a part of the three-dimensional model of the virtual robot (for example, a hand of a tool, etc.) may be changed by a gesture of the operator's hand or the like. In this case, for example, when the operator performs an action (hold action) of pinching the three-dimensional model displayed on the display, the pinched portion is specified as the operation target, and the position of the hand is determined. An operation to change the position or posture of the specified target may be performed by changing the position or posture, and the operation on the specified target may be ended by ending the pinching motion. In this case, for example, information indicating the target of the operation (for example, information indicating the position or part in the 3D model) and information indicating the content of the operation (for example, information indicating change in position or orientation). may be passed from the display device 4 to the reception unit 32, or the result of hand tracking acquired by the display device 4 may be passed to the reception unit 32, and the image processing device 3 may display the target of operation and the content of the operation. may be specified. When the display device 4 acquires the information indicating the target of the operation and the information indicating the content of the operation, the display device 4 displays the current three-dimensional image of the virtual robot in the virtual space held by the image processing device 3. Information in the dimensional model may be accessible. When an operation is performed using the operator's hand, the display device 4 has a camera, and hand tracking of the operator's hand photographed by the camera is performed. It may be specified where the hand is on the display. Further, a method of converting an operation to a display image of a three-dimensional model in response to a gesture of an operator's hand or the like to a change in the position or posture of the three-dimensional model in a three-dimensional virtual space is already known. Description is omitted. Further, the receiving unit 32 may receive, for example, an instruction to teach the three-dimensional model of the virtual robot, or an instruction to change the displayed three-dimensional model of the virtual robot.

For example, the receiving unit 32 may receive information input from an input device or the display device 4, or may receive information transmitted via a wired or wireless communication line. Note that the reception unit 32 may or may not include a device for reception (for example, an input device, a communication device, etc.). Further, the receiving unit 32 may be realized by hardware, or may be realized by software such as a driver for driving a predetermined device.

The positional relationship acquisition unit 33 acquires the relative positional relationship between the reference marker 6 and the display device 4 existing in the actual environment. Acquiring the relative positional relationship between the reference marker 6 and the display device 4 means, for example, determining the relative position between the marker coordinate system, which is the local coordinate system of the reference marker 6, and the display coordinate system, which is the local coordinate system of the display device 4. It may be to acquire a general positional relationship. This relative positional relationship may be indicated, for example, by a homogeneous transformation matrix that indicates transformation between both coordinate systems. The method by which the positional relationship acquisition unit 33 acquires this relative positional relationship does not matter. The positional relationship acquisition unit 33 receives an image captured by the camera of the display device 4, for example, and uses three or more feature points of the reference marker 6 included in the image to determine the relationship between the marker coordinate system and the display coordinate system. A homogeneous transformation matrix may be obtained that indicates the transformation of . Acquisition of the homogeneous transformation matrix may be performed in the display device 4 . In this case, the positional relationship acquisition unit 33 may receive from the display device 4 a homogeneous transformation matrix indicating transformation between the marker coordinate system and the display coordinate system. That is, the acquisition of the relative positional relationship by the positional relationship acquisition unit 33 may be reception of the relative positional relationship.

Based on the three-dimensional model of the virtual robot and the relative positional relationship between the reference marker 6 and the display device 4 acquired by the positional relationship acquisition unit 33, the image generation unit 34 determines a predetermined positional relationship between the reference marker 6 and the display device 4. A display image for displaying the three-dimensional model of the virtual robot is generated so that Also, when an operation to the virtual robot is accepted, the image generation unit 34 generates a display image for displaying the operated three-dimensional model. The predetermined positional relationship may be, for example, a positional relationship determined in advance, or a positional relationship that can be changed by the worker operating the virtual robot. The predetermined positional relationship may be, for example, a positional relationship in which the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6, or a position at which the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6. It can be a relationship. In the former case, the image generator 34 may generate a display image for displaying the three-dimensional model of the virtual robot at the position of the reference marker 6 . Displaying the three-dimensional model of the virtual robot at the position of the reference marker 6 means that the three-dimensional model of the virtual robot is displayed such that the situation in which the real robot is placed at the position of the reference marker 6 is virtually reproduced. For example, the three-dimensional model of the virtual robot may be constructed so that the end surface on the base end side (for example, the mounting surface on the floor surface) of the three-dimensional model of the virtual robot matches the surface of the reference marker 6. may be displayed. Displaying the three-dimensional model of the virtual robot at a position different from the reference marker 6 may mean, for example, displaying the three-dimensional model of the virtual robot next to the reference marker 6 . When the three-dimensional model of the virtual robot is displayed next to the reference marker 6, the figure 6b is normally displayed next to the reference marker 6 on the sheet 6a. In any case, the three-dimensional model of the virtual robot is displayed such that the positional relationship with the reference markers 6 placed in the real environment does not change. Therefore, even if the operator changes the orientation of the display device 4, the display position of the three-dimensional model of the virtual robot in the real environment does not change.

A relative positional relationship between the reference marker 6 and the display device 4 is acquired by the positional relationship acquisition unit 33 . Also, the positional relationship between the reference marker 6 and the three-dimensional model of the virtual robot is determined. Therefore, using these pieces of information, the image generator 34 can identify the relative positional relationship between the three-dimensional model of the virtual robot and the display device 4 . Therefore, the image generator 34 arranges the three-dimensional model of the virtual robot in the virtual space, and specifies the position and orientation of the display device 4 that has the specified positional relationship with respect to the three-dimensional model of the virtual robot. can be done. The image generator 34 then renders the three-dimensional model of the virtual robot in the virtual space based on the position and orientation of the display device 4, thereby generating a two-dimensional display image for displaying the three-dimensional model. be able to. Note that the angle of each joint of the virtual robot is the initial value when no operation is performed, and the post-operation value when the operation is performed. The angle of each joint after manipulation may be calculated, for example, by inverse kinematics using the position and orientation of the hand in the three-dimensional model of the virtual robot after manipulation, in the same way as with the real robot. When the virtual robot is manipulated, the shape of the three-dimensional model of the virtual robot in the virtual space is changed accordingly, as described above. Further, when the position and orientation of the display device 4 change in the real environment, the position and direction of the viewpoint in the virtual space are changed accordingly. By performing rendering after the change, a display image of the 3D model after the operation and a display image of the 3D model after the position and orientation of the display device 4 are changed are generated. Note that the image generation unit 34 generates a display image so that when the display image of the three-dimensional model of the virtual robot is displayed on the display of the display device 4, the size of the display image matches the actual environment. shall be That is, the three-dimensional model of the virtual robot displayed on the display of the display device 4 and the real robot arranged in the real environment so as to have the same relative positional relationship as the three-dimensional model are displayed via the display device 4. A display image is generated so that it has the same size when viewed from above.

Further, for example, when the reception unit 32 receives an instruction to change the three-dimensional model of the virtual robot to be displayed, the image generation unit 34 generates the three-dimensional model of the virtual robot to be displayed according to the instruction. You can change the dimensional model. For example, when a plurality of 3D models of virtual robots are stored in the storage unit 31, the image generation unit 34, in response to an instruction to change the display target, creates a virtual robot different from the previous one. A display image may be generated using the three-dimensional model. If the instruction includes information for identifying the three-dimensional model of the virtual robot to be displayed, the display image may be generated using the three-dimensional model of the virtual robot identified by the information.

Also, it may not be possible to operate the virtual robot according to the accepted operation. For example, an operation to move the hand of the virtual robot beyond its movable range or an operation to rotate it beyond its rotatable range may be accepted. In such a case, the image generator 34 does not have to generate a display image corresponding to the operation, or a display image of the three-dimensional model of the virtual robot that has moved or rotated within the possible range. may be generated.

The output unit 35 outputs the display image generated by the image generation unit 34 to the display device 4 . Note that the output unit 35 may output only the display image. In this case, on the display device 4, the display image is superimposed on the image of the real environment or the real environment itself. On the other hand, when the display device 4 has a non-transmissive display and the image processing device 3 receives an image of the real environment photographed by the display device 4, the image of the real environment and the display image are may be output to the display device 4 . This synthesis may be performed by, for example, a synthesis unit (not shown) included in the image processing device 3 .

When the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6, for example, as shown in FIG. 1, the three-dimensional model 10 of the virtual robot may be displayed next to the reference marker 6. . The reference marker 6 and the three-dimensional model 10 of the virtual robot shown in FIG. In this case, for example, the graphic 6b may not be displayed on the sheet 6a.

Displaying a three-dimensional model of a virtual robot and manipulating the three-dimensional model of the virtual robot using a virtual input interface of a display device, gestures of a worker, etc. are disclosed, for example, in the following patent documents: As indicated by 1 to 3, they are already known, and detailed description thereof will be omitted.

JP 2017-100234 A JP 2020-055075 A JP 2020-069538 A

Next, the operation of the image processing device 3 will be described using the flowchart of FIG.

(Step S101) The accepting unit 32 determines whether or not an operation to the three-dimensional model of the virtual robot has been accepted. Then, if the operation is accepted, the process proceeds to step S103, and if not, the process proceeds to step S102.

(Step S102) The image generator 34 determines whether to generate a display image. If a display image is to be generated, the process proceeds to step S103; otherwise, the process returns to step S101. Note that the image generator 34 may periodically determine, for example, to generate the display image. By making this judgment, for example, even if no operation is performed, if the position or orientation of the display device 4 is changed, the display image corresponding to the changed position or orientation is displayed on the display device 4 . will be displayed.

(Step S<b>103 ) The positional relationship acquisition section 33 acquires the relative positional relationship between the reference marker 6 and the display device 4 .

(Step S104) Using the relative positional relationship obtained in step S103 and the three-dimensional model of the virtual robot, the image generator 34 creates a three-dimensional model so as to have a predetermined positional relationship with the reference marker 6. Generate a display image for display. Note that when the operation is accepted, a display image for displaying the three-dimensional model after being changed according to the operation is generated. Further, when the position and orientation of the display device 4 are changed, the display image of the three-dimensional model is generated according to the changed position and orientation of the display device 4 .

(Step S<b>105 ) The output unit 35 outputs the generated display image to the display device 4 . Then, the process returns to step S101. According to this output, a three-dimensional model of the virtual robot is displayed so as to have a predetermined positional relationship with the reference marker 6 .

The order of processing in the flowchart of FIG. 2 is an example, and the order of each step may be changed as long as the same result can be obtained. In addition, in the flowchart of FIG. 2, the processing ends when the power is turned off or an interrupt for processing end occurs.

Next, the method of arranging the real robots will be described with reference to the flowchart of FIG.

(Step S201) The operator places the reference marker 6, which is a predetermined two-dimensional image, in the real environment. The reference marker 6 may, for example, be arranged at a planned position for arranging the real robot. Also, for example, when another real robot is placed at the position where the real robot is to be placed, the robot may be placed next to the existing real robot.

(Step S<b>202 ) The operator uses the image processing device 3 and the display device 4 to operate the three-dimensional model of the virtual robot displayed on the display device 4 so as to have a predetermined positional relationship with the reference marker 6 . The purpose of this operation is, for example, to check the reachable range of the robot, to check whether the robot can assume a desired posture, or to check whether the robot can avoid existing obstacles. may be done for It should be noted that the three-dimensional model of the virtual robot may be displayed at the position of the reference marker 6 when the reference marker 6 is arranged at the position where the real robot is to be arranged. In addition, when the reference marker 6 is placed next to the existing real robot, the three-dimensional model of the virtual robot is placed at a new robot placement position that has a predetermined positional relationship with the reference marker 6, for example, the existing robot. may be displayed at the position where the real robot is arranged.

(Step S203) The operator determines whether the arrangement of the virtual robots and the type of the virtual robots are appropriate. For example, if the reachable range of the robot is not a problem, the robot can assume a desired posture, and the robot can avoid existing obstacles, the worker will feel that the placement and type of the virtual robot are appropriate. You can judge. On the other hand, if there is a problem with the reachable range of the robot, if the robot cannot assume the desired posture, or if the robot cannot avoid existing obstacles, the worker will be able to determine the placement or type of virtual robot. may be judged to be inappropriate. Then, if the placement of the virtual robot, that is, the placement of the reference marker 6 is not appropriate, the process returns to step S201 and the placement of the reference marker 6 is redone. Also, if the type of virtual robot is not appropriate, the type of virtual robot may be changed and the operation may be performed again. In this case, the virtual robot type is changed, the process returns to step S202, and the new virtual robot is operated. On the other hand, if the placement and type of virtual robot are appropriate, the process proceeds to step S204.

(Step S204) After manipulating the three-dimensional model of the virtual robot, the real robot is arranged so as to have a predetermined positional relationship with the reference markers 6 arranged in the real environment. This real robot corresponds to the three-dimensional model of the manipulated virtual robot. For example, when a figure 6b for specifying the placement position of the real robot is displayed on the sheet 6a on which the reference marker 6 is displayed, a screw hole for arranging the real robot is displayed at the position of the figure 6b. may be provided, and the actual robot may be fixed to the floor or the like using the screw holes. In this way, the real robot can be arranged at the same position as the virtual robot used for checking the motion, and the real robot can be arranged at an appropriate position.

Although not included in the flowchart of FIG. 3, teaching using a virtual robot may be performed. In this case, for example, in a situation where the reference marker 6 is placed at an appropriate position, that is, a situation where the virtual robot is displayed at the same position as the real robot to be introduced, the three-dimensional model of the virtual robot is operated, and a teaching instruction may be input as appropriate. Teaching data may be stored in the storage unit 31 in accordance with the teaching instruction. Also, the teaching data may be passed to a robot control device that controls the real robot. After the real robot is placed using the reference marker 6, the real robot may be operated using the teaching data. In this case, the teaching data can be prepared in advance before the introduction of the real robot, and the playback operation using the teaching data can be started immediately after the introduction of the real robot.

Next, the robot placement method and the operation of the image processing apparatus 3 according to this embodiment will be described using a specific example. In this specific example, it is assumed that a three-dimensional model of a virtual robot corresponding to the robot 1, which is a real robot to be introduced, is stored in the storage unit 31 of the image processing device 3 in advance.

First, as shown in FIG. 4A, the operator attaches the sheet 6a on which the reference marker 6 is displayed to the position where the robot 1, which is the actual robot to be introduced, is placed (step S201). It is assumed that four figures 6b for specifying the layout position of the robot 1 are displayed on the sheet 6a.

Next, the operator wears the head-mounted display device 4 on the head and operates the three-dimensional model 10 of the virtual robot displayed by the image processing device 3 (step S202). Specifically, when the image generator 34 determines to generate a display image (step S102), the relative positional relationship between the reference marker 6 and the display device 4 is obtained (step S103) and stored. A display image for displaying the three-dimensional model is generated using the three-dimensional model stored in the unit 31 and the relative positional relationship, and is output to the display device 4 (steps S104 and S105). In this way, a three-dimensional model 10 of the virtual robot is displayed at the position of the reference marker 6, as shown in FIG. 4B.

In addition, when the worker operates the three-dimensional model of the virtual robot using a teaching pendant, a virtual input interface, or a gesture operation using a hand, a display image corresponding to the operation is generated and displayed (step S101, S103-S105). This operation may be, for example, a hand operation of the virtual robot. Note that the hand of the virtual robot may be, for example, a TCP (tool center point). In this way, the operator can use the virtual robot to understand the conditions under which the robot is placed, such as the reachable range of the robot, whether the robot can assume a desired posture, and whether the robot can overcome existing obstacles. You can check whether or not you can avoid objects. Then, if the placement position of the robot is not appropriate, the sticking position of the sheet 6a displaying the reference marker 6 is changed, and the placement situation of the robot is confirmed again. Also, if the type of robot is not appropriate, the type of virtual robot to be displayed is changed, and the arrangement of robots is checked again.

If there is no problem with the arrangement and type of the virtual robot (step S203), the operator ends the display of the virtual robot using the image processing device 3 and the display device 4, and moves the sheet 6a on which the reference marker 6 is displayed. Bolt holes for fixing robots 1, which are real robots, are provided at the positions of figures 6b. Then, as shown in FIG. 4C, the robot 1 to be introduced is placed at the position of the reference marker 6, and the proximal end of the robot 1 is fixed to the bolt hole with the bolt 1a (step S204). In this way, the robot 1, which is a real robot, can be placed at a position whose motion has been confirmed in advance by the virtual robot.

As described above, according to the image processing apparatus 3 and the robot placement method according to the present embodiment, it is possible to check the placement status of the robot to be introduced using the virtual robot before placement of the real robot. Therefore, it is possible to place the actual robot after confirming the placement position of the robot to be introduced and the type of robot to be introduced in advance, so that unexpected situations do not occur after the actual robot is placed. can be Moreover, when the reference marker 6 is a two-dimensional image, the display position of the virtual robot can be easily changed according to the change of the arrangement position of the reference marker 6 in the real environment. In addition, when the reference marker 6 is displayed on the sheet 6a and the figure 6b for specifying the arrangement position of the real robot is also displayed on the sheet 6a, the real robot can be displayed by using the figure 6b. can be placed at the same position as the display position of the virtual robot whose motion has been confirmed.

In this embodiment, the case where the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6 has been mainly described, but this need not be the case. For example, a three-dimensional model of the virtual robot may be displayed at a predetermined position of the reference marker 6 in the marker coordinate system. Also, a three-dimensional model of the virtual robot may be displayed at a predetermined position and a predetermined orientation in the marker coordinate system. In this case, it is assumed that the actual robot is arranged so as to have a predetermined position and a predetermined posture with respect to the reference marker 6 when arranging the actual robot.

Also, in the present embodiment, the case where the three-dimensional model of the virtual robot is displayed after the reference marker 6 is placed in the real environment has been mainly described, but this need not be the case. For example, the reference marker 6 may be placed after a three-dimensional model of the virtual robot is displayed and the operator moves the displayed position to a desired position. In this case, even if the positional relationship between the reference marker 6 and the three-dimensional model of the virtual robot when the reference marker 6 is arranged becomes a predetermined positional relationship between the reference marker and the three-dimensional model of the virtual robot. good. In addition, until the reference marker 6 is arranged, the positional relationship acquisition unit 33 acquires the relative positional relationship between a predetermined object existing in the real environment and the display device 4, and uses the relative positional relationship to Alternatively, the three-dimensional model of the virtual robot may be displayed such that the positional relationship between the three-dimensional model of the virtual robot and a predetermined object is constant. It should be noted that the positional relationship between the three-dimensional model of the virtual robot and the predetermined object can be appropriately changed by the operator's operation. The predetermined object may be, for example, a real robot that exists in the real environment, a jig, or any other object.

Also, in the present embodiment, the type of virtual robot to be displayed may be specified by the reference marker 6 . For example, when the reference marker 6 is a two-dimensional code such as an AR marker or a QR code (registered trademark), the identifier of the two-dimensional code is associated with the type of virtual robot, and the image generator 34 displays the An image of the reference marker 6 or an identifier of the two-dimensional code that is the reference marker 6 is received from the device 4, and a display image is generated using a three-dimensional model of the virtual robot corresponding to the identifier of the two-dimensional code that is the reference marker 6. may In this case, the operator has a plurality of sheets 6a on which reference markers 6 are displayed for each virtual robot, and by appropriately changing the sheet 6a arranged in the real environment, the virtual robot to be displayed can be displayed. can be changed easily. The identifier of the two-dimensional code may be, for example, information obtained by reading the two-dimensional code, or may be an identifier associated with the two-dimensional code.

Also, in the present embodiment, the case where the reference marker 6 is a predetermined two-dimensional image has been mainly described, but it may not be so. Similar to markerless AR, objects existing in the real environment may be used as reference markers. Objects that exist in the real environment are not particularly limited, but may be, for example, a base part used for installing the real robot, a jig that exists near the position where the robot is arranged, or the like. Also, a marker on a three-dimensional object (for example, a rectangular parallelepiped) having a marker attached to each surface thereof may be used as the reference marker. Also, the reference marker may be three-dimensional, such as a base or a jig.

Further, while the three-dimensional model of the virtual robot is being operated, the image processing device 3 may perform collision determination between the three-dimensional model and an object existing in the real environment. By performing such a collision determination, the worker operating the virtual robot can easily know whether or not at least a part of the three-dimensional model of the virtual robot has interfered with an obstacle in the real environment. become able to.

Further, in the above embodiments, each process or function may be implemented by centralized processing by a single device or single system, or may be implemented by distributed processing by multiple devices or multiple systems. It may be realized by For example, at least part of the configuration of the image processing device 3 may be physically included in a device having a display. Therefore, it may be considered that the division of devices shown in FIG. 1 is for the sake of convenience according to the functions rather than according to the physical devices.

Further, in the above-described embodiment, when the information is passed between the components, for example, when the two components that exchange the information are physically different, one of the components output of information and reception of information by the other component, or one component if the two components that pass the information are physically the same from the phase of processing corresponding to the other component to the phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component Also, information such as thresholds, formulas, addresses, etc. used by each component in processing may be stored temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, each component or an accumulation section (not shown) may accumulate information in the recording medium (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

Further, in the above embodiment, if the information used in each component etc., for example, information such as thresholds, addresses and various set values used in processing by each component may be changed by the user, the above The user may or may not be able to change such information as appropriate, even if not explicitly stated in the description. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives a change instruction from the user and a change unit (not shown) that changes the information according to the change instruction. may The reception of the change instruction by the reception unit (not shown) may be, for example, reception from an input device, reception of information transmitted via a communication line, or reception of information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the image processing apparatus 3 have communication devices, input devices, etc., the two or more components may physically have a single device. or may have separate devices.

Further, in the above embodiments, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU. During the execution, the program execution unit may execute the program while accessing the storage unit or recording medium. Further, the program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). good. Also, this program may be used as a program constituting a program product. Also, the number of computers that execute the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible and are also included within the scope of the present invention.

3 image processing device, 4 display device, 6 reference marker, 31 storage unit, 32 reception unit, 33 positional relationship acquisition unit, 34 image generation unit, 35 output unit, 100 robot control system

Claims (5)

a storage unit in which a three-dimensional model of a virtual robot is stored;
a reception unit that receives an operation on the three-dimensional model of the virtual robot;
a positional relationship acquiring unit that acquires a relative positional relationship between a reference marker that exists in the real environment and a display device that displays an image superimposed on the image of the real environment or the real environment itself;
an image generator for generating a display image for displaying the manipulated three-dimensional model so as to have a predetermined positional relationship with the reference marker based on the three-dimensional model and the relative positional relationship;
and an output unit that outputs the display image to a display device. 2. The image processing apparatus according to claim 1, wherein said reference marker is a predetermined two-dimensional image. 3. The image processing apparatus according to claim 2, wherein said reference marker is displayed on a sheet, and said sheet displays a figure for specifying the placement position of said real robot. 4. The image processing apparatus according to claim 1, wherein said image generator generates a display image for displaying a three-dimensional model of said virtual robot at the position of said reference marker. placing fiducial markers, which are two-dimensional predetermined images, in the real environment;
a step of operating a three-dimensional model of a virtual robot displayed so as to have a predetermined positional relationship with the reference marker on a display device that displays an image superimposed on an image of the real environment or the real environment itself;
and a step of arranging a real robot corresponding to the 3D model of the virtual robot so as to have a predetermined positional relationship with the reference marker in a real environment after manipulating the 3D model.

Images