JP2023017441A - Image processing device - Google Patents

Abstract

To solve the problem that is difficult to intuitively perform operation to a virtual robot, because an operator cannot sense touch to the virtual robot.SOLUTION: An image processing device 3 comprises: a storing part 31 that stores a three-dimensional model of a virtual robot; an operation obtaining part 34 that obtains a position of an operation instructing member existing in an actual environment and obtains operation to a portion corresponding to the operation instructing member in the three-dimensional model; a positional relation obtaining part 32 that obtains a relative positional relation between the actual environment and a display device 4 that displays an image while superimposing the image on an image of the actual environment or on the actual environment; an image generating part 35 that generates a display image for displaying the operated three-dimensional model, on a predetermined position in the actual environment, on the basis of the three-dimensional model and the relative positional relation; and an image output part 36 that outputs the display image to the display device 4. Thus, the virtual robot can be operated using the operation instructing member in the actual environment, which enables operation to the virtual robot to be performed more intuitively.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus that generates a display image of a three-dimensional model of a virtual robot to be operated.

A worker sometimes manipulates a real robot by manipulating a three-dimensional model of a virtual robot through gesture manipulation (see, for example, Patent Document 1). Then, for example, direct teaching in the virtual space may be performed by the operator pushing or grasping and moving a tool at the end of the three-dimensional model or other parts with a finger. .

JP 2020-055075 A

However, when performing an operation using a three-dimensional model of a virtual robot, there is a problem that it is difficult to perform an intuitive operation because the operator who is performing the operation does not have a sense of touch. For example, when direct teaching is performed by moving a tool of a real robot by hand, the operator feels as if he or she is grasping the tool by hand. It is possible to confirm whether or not the expected change has been realized by the feeling of the hand. On the other hand, when manipulating a 3D model in virtual space, it is not possible to get the feeling of grasping it. There was a problem that it was difficult to judge whether the expected changes were realized. Moreover, as a result, the result of the operator's operation may not be as expected.

The present invention has been made to solve the above-mentioned problems, and even when a worker operates a three-dimensional model of a virtual robot in a virtual space, the operation is more similar to that of a real robot. It is an object of the present invention to provide an image processing apparatus capable of

To achieve the above object, an image processing apparatus according to an aspect of the present invention includes a storage unit storing a three-dimensional model of a virtual robot corresponding to a real robot existing in the real environment, and an operation instruction member existing in the real environment. and acquires an operation for a location corresponding to the operation instruction member in the three-dimensional model of the virtual robot according to the position, a real environment where the real robot exists, and an image of the real environment. At a predetermined position in the real environment based on the positional relationship acquisition unit that acquires the relative positional relationship with the display device that is superimposed on the image or the real environment itself, and the relative positional relationship with the three-dimensional model, The apparatus includes an image generation unit that generates a display image for displaying an operated three-dimensional model, and an image output unit that outputs the display image to a display device.

According to an image processing apparatus according to an aspect of the present invention, an operation of a virtual robot on a three-dimensional model is obtained in accordance with an operation performed by a worker using an operation instruction member existing in a real environment, and display image can be generated. Therefore, since the operator can operate the three-dimensional model of the virtual robot by changing the position and posture of the operation instruction member, it is possible to operate the three-dimensional model of the virtual robot in the virtual space, while the operator can operate the three-dimensional model of the virtual robot in a virtual space. You will be able to get a feeling close to operating the . As a result, it is possible to reduce the possibility that the operator's operation will result in an unexpected result.

Schematic diagram showing the configuration of a robot control system according to an embodiment of the present invention. 3 is a flow chart showing the operation of the image processing apparatus according to the same embodiment; A diagram for explaining an example of an operation in the same embodiment. A diagram for explaining an example of an operation in the same embodiment. A diagram for explaining an example of an operation in the same embodiment. A diagram showing an example of the relationship between an operation instruction member and a three-dimensional model in the same embodiment. A diagram showing an example of the relationship between an operation instruction member and a three-dimensional model in the same embodiment.

An image processing apparatus according to the present invention will be described below using embodiments. In the following embodiments, constituent elements and steps with the same reference numerals are the same or correspond to each other, and repetitive description may be omitted. The image processing apparatus according to the present embodiment acquires the operation of the virtual robot on the three-dimensional model according to the position of the operation instruction member and the like.

FIG. 1 is a schematic diagram showing the configuration of a robot control system 100 according to this embodiment. A robot control system 100 according to this embodiment is for operating a virtual robot, and includes an image processing device 3 and a display device 4 . Note that the image processing device 3 and the display device 4 may be connected by wire or wirelessly, for example. Further, when the robot 1, which is a real robot, is operated by operating the three-dimensional model of the virtual robot, the robot control system 100 includes, for example, the robot 1 and a robot control system for controlling the motion of the robot 1. and a device. Manipulation of the robot 1 via the three-dimensional model of the virtual robot may be performed, for example, for teaching purposes or for confirmation of reachable ranges and the like. When the robot 1 is manipulated by manipulating the three-dimensional model, for example, the robot 1 may be manipulated so that the angles of the joints of the robot 1 are the same as the angles of the joints of the three-dimensional model. good.

The virtual robot to be operated is configured by a three-dimensional model that exists in the virtual environment, and corresponds to the real robot that exists in the real environment. That is, the virtual robot is the same as the real robot, except that it is configured as a three-dimensional model. For example, it may have the same size and configuration as the real robot. , etc. may be changeable.

Real robots are typically industrial robots and may be manipulators with multiple arms (links) connected by motor-driven joints. The real robot may be, for example, a vertical articulated robot or a horizontal articulated robot. Further, the actual robot may be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. good too. A real robot is a robot that exists in a real environment. The real environment is the environment of the real space. The robot control device controls the motion of the robot 1 . A robot control device that operates the robot 1 is already known, and a detailed description thereof will be omitted. For example, when the robot control system 100 includes the robot 1, which is a welding robot, the robot control system 100 may further include a welding power supply, a wire feeder, and the like as necessary.

The image processing device 3 generates a display image for displaying a three-dimensional model of the virtual robot at a predetermined position in the real environment, and outputs the display image to the display device 4 . The display image of the three-dimensional model of the virtual robot may be generated so that the three-dimensional model of the virtual robot and the robot 1, which is a real robot existing in the real environment, have a predetermined positional relationship. Details of the image processing device 3 will be described later.

The display device 4 displays the image superimposed on the image of the real environment or the real environment itself. In other words, the operator who operates the virtual robot can see both the images of the real environment and the virtual environment on the display device 4 . The display device 4 may be a wearable display device worn on the head by the operator who operates the virtual robot, or may be a display device that is a portable information processing terminal such as a tablet terminal. . The wearable display device may be, for example, a head-mounted display. Moreover, the display device 4 may have, for example, a transmissive display. In this case, the display device 4 displays the image superimposed on the real environment itself. HoloLens (registered trademark), for example, is known as a wearable display device 4 having a transmissive display. The display device 4 having such a transmissive display can also be considered as a display device for realizing mixed reality (MR). The display device 4 may also have, for example, a non-transmissive display. In this case, the display device 4 displays the image superimposed on the image of the real environment. Therefore, the display device 4 having a non-transmissive display preferably has a camera for photographing the real environment or is connected to a camera for photographing the real environment. Images of the real environment captured by the camera are displayed in real time on the non-transmissive display. Oculus Quest, for example, is known as a wearable display device 4 having a non-transmissive display. The display device 4 having such a non-transmissive display can also be considered as a display device for realizing augmented reality (AR). The display device 4, which is a portable information processing terminal such as a tablet terminal, has, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. In this embodiment, the case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

As shown in FIG. 1, the image processing apparatus 3 according to the present embodiment includes a storage unit 31, a positional relationship acquisition unit 32, a reception unit 33, an operation acquisition unit 34, an image generation unit 35, and an image output unit. a portion 36;

The storage unit 31 stores a three-dimensional model of a virtual robot corresponding to a real robot existing in the real environment. This three-dimensional model of the virtual robot may be, for example, a three-dimensional model of the virtual robot corresponding to the real robot to be operated. A three-dimensional model of a virtual robot may be composed of, for example, a plurality of partial objects. Each partial object may correspond to each arm or hand tool of the three-dimensional model of the manipulator. Information other than the three-dimensional model may also be stored in the storage unit 31 . For example, teaching data or the like may be stored in the storage unit 31 . The process by which information is stored in the storage unit 31 does not matter. For example, information may be stored in the storage section 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage section 31 . The storage unit 31 is preferably implemented by a non-volatile recording medium, but may be implemented by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment in which the real robot exists and the display device 4 . For example, a robot 1, which is a real robot, may exist in the real environment. Acquiring the relative positional relationship between the real environment and the display device 4 means, for example, obtaining the relative position between the world coordinate system, which is the coordinate system of the real environment, and the display coordinate system, which is the local coordinate system of the display device 4. It may be to obtain a relationship. The relative positional relationship may be indicated, for example, by a homogeneous transformation matrix that indicates transformation between both coordinate systems. The relative positional relationship between the real environment and the display device 4 may be, for example, the relative positional relationship between the robot 1 or other objects existing in the real environment and the display device 4. may be information indicating the position and orientation of the display device 4 in .

The method by which the positional relationship acquisition unit 32 acquires this relative positional relationship does not matter. For example, when an environment map of the real environment is prepared, the positional relationship acquisition unit 32 uses the environment map of the real environment and the distance to the surrounding objects and the image of the surroundings acquired by the display device 4. Alternatively, SLAM (Simultaneous Localization and Mapping) or Visual-SLAM techniques may be used to obtain the relative positional relationship, which is information indicating the position and orientation of the display device 4 in the three-dimensional real environment. In this case, for example, an environment map is held in the image processing device 3, and the positional relationship acquisition unit 32 uses the distance to the surrounding objects and the surrounding image received from the display device 4, and the environment map. to obtain the relative positional relationship. Further, for example, acquisition of the relative positional relationship using the SLAM or Visual-SLAM technique may be performed in the display device 4 and the relative positional relationship may be passed to the positional relationship acquiring section 32 . In this case, the acquisition of the relative positional relationship by the positional relationship acquiring unit 32 may be reception of the relative positional relationship. Also, in this case, the display device 4 may have a depth sensor capable of measuring the distance to surrounding objects and a camera capable of acquiring surrounding images. In addition, the positional relationship acquisition unit 32 receives an image captured by the camera of the display device 4, and uses three or more feature points of the robot 1 and other objects included in the image to determine the world coordinate system and the display coordinate system. A homogeneous transformation matrix may be obtained that indicates the transformation between and .

Further, the positional relationship acquisition unit 32 may acquire the relative positional relationship between the real environment and the display device 4 by a method other than the above. For an example of the method, see, for example, the above-mentioned Patent Document 1 and the like. Also, the relative positional relationship may or may not be acquired by the positional relationship acquiring unit 32 as described above each time a display image to be described later is generated. In the latter case, for example, even if the relative positional relationship between the real environment and the display device 4 is updated using changes in the position and orientation of the display device 4 after the relative positional relationship is obtained, good. This update may be performed by the positional relationship acquisition unit 32, for example. Further, when this update is performed, the display device 4 is provided with, for example, an acceleration sensor and a sensor for acquiring the orientation of the display device 4, and these sensors are used to detect changes in position and orientation. Changes may be obtained. Note that the values acquired by those sensors may be passed to a component that updates the relative positional relationship, such as the positional relationship acquiring unit 32, for example. A sensor for acquiring the orientation of the display device 4 may be, for example, a gyro sensor, an orientation sensor, or the like.

The accepting unit 33 accepts a start instruction for instructing the start point of an operation using the operation instruction member 5 and an end instruction for instructing the end point of the operation. The start instruction and the end instruction may be input by voice such as “start operation” and “end operation”, or may be input via an input device or a virtual input interface of the display device 4 .

When a voice input is performed, the reception unit 33 receives a voice corresponding to an utterance such as "operation start" or "operation end" by the operator with a microphone or the like, and converts the voice into a signal for voice recognition. may be performed, and the start instruction may be determined to have been received when the text information, which is the result of the voice recognition, is "operation start". The same applies to the end instruction. Note that the reception unit 33 may receive a start instruction or an end instruction corresponding to the result of voice recognition from another component or another device.

Also, the input device may have, for example, a physical button for inputting a start instruction and a physical button for inputting an end instruction. Then, for example, when the worker presses a physical button for inputting the start instruction, the reception unit 33 may determine that the start instruction has been received. The same applies to the end instruction.

Further, the operator may display a virtual input interface such as a virtual button on the display of the display device 4 according to an action such as air tapping. Then, for example, when a virtual button for inputting a start instruction is selected by an operator's finger or a pointing device, and the fact that the virtual button has been selected is input to the receiving unit 33, the receiving unit 33 It may be determined that the start instruction has been received. The same applies to the end instruction.

Note that the reception unit 33 may or may not include a device for reception (for example, a microphone, an input device, a communication device, etc.). Further, the reception unit 33 may be realized by hardware, or may be realized by software such as a driver for driving a predetermined device.

The operation acquisition unit 34 acquires the position of the operation instruction member 5 existing in the actual environment. The operation acquisition unit 34 may acquire the position and orientation of the operation instruction member 5, for example. Although the shape of the operation instruction member 5 is not particularly limited, it may be, for example, a rod-shaped member or a member having other shapes. The rod-shaped member may be, for example, a columnar member or a polygonal columnar member. Further, as shown in FIG. 1, the operation instruction member 5, which is a rod-shaped member, has different shapes at both ends in the longitudinal direction, such as one end being pointed, so that the orientation can be determined. It can be like this. As the operation instruction member 5, for example, a writing implement such as a pen or a pencil may be used. In this embodiment, the case where the operation instructing member 5 is a rod-shaped member will be mainly described, and other cases will be described later. The operation instructing member 5 is usually moved manually by the operator. The operation instruction member 5 is, for example, about the same size as a writing instrument such as a pen, and may be held and moved by the operator's hand in the same manner as the writing instrument. The position and orientation of the operation instruction member 5 may indicate, for example, the position and orientation of a tool attached to the hand of the three-dimensional model of the virtual robot, and the TCP (tool center point) of the three-dimensional model of the virtual robot. The position and orientation may be indicated, and the position and orientation of a predetermined portion (for example, a predetermined arm, etc.) in the three-dimensional model of the virtual robot may be indicated. The location of the three-dimensional model corresponding to the operation instructing member 5 may be determined in advance, or may be determined by the operator. In this embodiment, the case where the position and orientation of the operation instructing member 5 indicates the position and orientation of the hand tool of the three-dimensional model will be mainly described.

Any method may be used by which the operation acquisition unit 34 acquires the position and orientation of the operation instruction member 5 . Acquiring the position and orientation of the operation instruction member 5 may be, for example, specifying the position and orientation of the operation instruction member 5 in a predetermined coordinate system. By repeating the acquisition of the position and orientation in real time, the movement of the operation instructing member 5 is acquired as a result. The coordinate system may be, for example, the world coordinate system, the local coordinate system of the display device 4, or any other coordinate system. When the operation acquisition unit 34 acquires the position of the operation instruction member 5, for example, a predetermined portion of the operation instruction member 5 (for example, the tip on the lower side of the operation instruction member 5 shown in FIG. 1) You can get the position. This position may be indicated, for example, by coordinate values in a three-dimensional orthogonal coordinate system. Further, when the operation acquisition unit 34 acquires the posture of the operation instruction member 5, for example, the angle of the rod-shaped operation instruction member 5 in the longitudinal direction and the longitudinal axis of the rod-shaped operation instruction member 5 are taken as the center. and a rotation angle of . This attitude may also be indicated by, for example, roll, pitch and yaw angles, Euler angles, and the like. Note that when the rotation angle about the longitudinal axis of the rod-shaped operation instruction member 5 is also acquired, or when the angles of the three degrees of freedom according to the posture are acquired, the operation instruction member 5 A surface (for example, the circumferential surface of the operation instruction member 5 having a cylindrical shape) is provided with a predetermined pattern, protrusions, recesses, or the like. You may come to know an angle. In the present embodiment, a case where the operation acquisition unit 34 acquires both the position and orientation of the operation instruction member 5 will be mainly described.

The operation acquisition unit 34 may acquire the position and the like using, for example, a photographed image of the operation instruction member 5 . In this case, the image may be captured using, for example, a camera included in the display device 4, or may be performed using a camera arranged at a predetermined position in the real environment. When a camera for photographing the actual environment is attached to the head-mounted display device 4, the camera obtains a photographed image in the same direction as the line of sight of the worker wearing the display device 4. It is preferred that they are arranged as follows. Moreover, when a camera is arranged in the real environment, it is preferable that the camera be arranged so as to be able to photograph the area where the operation instruction member 5 is moved by the operator. The operation acquisition unit 34 may, for example, identify and track the operation instruction member 5 using the captured image, and acquire the position and the like of the operation instruction member 5 using the results.

Further, the operation acquisition unit 34 may acquire the position or the like using sensing results from a sensor incorporated in the operation instruction member 5, for example. The sensor may be, for example, an acceleration sensor or a sensor for acquiring the orientation of the operation instruction member 5 . The sensors for acquiring the attitude may be, for example, an orientation sensor for acquiring an azimuth angle, an inclination sensor for acquiring an inclination angle with respect to the vertical direction, or a gyro sensor (that is, an angular velocity sensor). may Since the acceleration sensor and the gyro sensor only detect changes in position and angle, for example, the operator may first place the operation instruction member 5 at a predetermined reference position and reference attitude. Then, the operation acquisition unit 34 may acquire the current position and orientation of the operation instruction member 5 using the sensing result acquired by the acceleration sensor and the gyro sensor, the reference position, and the reference orientation. The sensing result of the sensor of the operation instruction member 5 may be transmitted wirelessly or by wire from the operation instruction member 5 to the operation acquisition unit 34 .

In addition, the operation acquisition unit 34 uses a sensing result indicating the position of the operation instruction member 5 acquired by, for example, a sensor arranged in the actual environment or a sensor included in the display device 4 to determine the position of the operation instruction member 5 and the like. may be obtained. This sensor may be, for example, a ranging sensor, such as a depth sensor. When the sensors are ranging sensors, multiple ranging sensors may be used to perform a three-dimensional scan of the real environment. Then, the operation acquiring unit 34 may specify the position and orientation of the operation instructing member 5 having a predetermined shape using the sensing result, that is, the three-dimensional scanning result.

Further, acquisition of the position of the operation instruction member 5 using the captured image or the sensing result may be performed by a component or device other than the operation acquisition unit 34 . In this case, the operation acquisition unit 34 may receive the position and orientation of the operation instruction member 5 acquired by a component or device other than the operation acquisition unit 34 . That is, the acquisition of the position and orientation of the operation instruction member 5 by the operation acquisition unit 34 may be reception of the position and orientation of the operation instruction member 5 .

The operation acquisition unit 34 acquires an operation for a location corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot, according to the acquired position and orientation of the operation instruction member 5 . The location corresponding to the operation instruction member 5 may be a location where the operation is indicated by the operation instruction member 5 . The location corresponding to the operation instruction member 5 may be, for example, a location in the three-dimensional model whose position and orientation are indicated by the operation instruction member 5 . For example, an operation for causing a portion corresponding to the operation instruction member 5 in the three-dimensional model to overlap the operation instruction member 5 may be acquired. Acquisition of this operation may be, for example, acquisition of the post-operation position and orientation of the portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot. For example, if the operation instruction member 5 is a rod-shaped member that indicates the position and orientation of a tool attached to the hand of the three-dimensional model of the virtual robot, the operation acquisition unit 34 may obtain the operation instruction member 5 as Depending on the position or the like, an operation of the three-dimensional model tool of the virtual robot may be obtained. That is, the operation acquisition unit 34 may acquire an operation in which the three-dimensional model tool is positioned at the operation instruction member 5 . The tools are arranged at the tip of the three-dimensional model of the virtual robot, and include, for example, a welding torch, a gripping unit that grips an object to be transported, a suction unit that adsorbs an object to be transported, and an object to be transported. A placement unit or the like may be used. For example, when the tool is a welding torch, the rod-shaped operation indicating member 5 may indicate the position and posture of the tip of the welding torch. In this case, the operation acquisition unit 34 acquires an operation for changing the position and orientation of the welding torch in the same manner as the position and orientation of the operation instruction member 5 . Further, for example, when the tool is a grasping unit, a suction unit, or the like, the rod-shaped operation instructing member 5 may indicate the reference position and orientation of the unit in the reference direction. In this case, the operation acquiring unit 34 acquires an operation for changing the reference position and orientation of the gripping unit and the suction unit in the same manner as the position and orientation of the operation instructing member 5 . It will be.

Note that when the position and orientation of the operation instruction member 5 are the position and orientation of the location corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot, the operation acquisition unit 34 obtains the position and orientation of the operation instruction member 5. And the posture may be an operation for a portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot. In this embodiment, this case will be mainly described. Note that, even in this case, the operation acquisition unit 34, for example, obtains the position and orientation of the operation instruction member 5 in the display coordinate system, which is obtained by the positional relationship acquisition unit 32. Using the homogeneous transformation matrix and the homogeneous transformation matrix between the world coordinate system and the local coordinate system of the three-dimensional model of the virtual robot, the results of transformation into the position and orientation in the local coordinate system of the virtual robot are transformed into the virtual robot. It may be an operation to a three-dimensional model. On the other hand, if the position and orientation of the operation instruction member 5 are different from the position and orientation of the portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot, the operation acquisition unit 34 determines the position and orientation of the operation instruction member 5. and the orientation, the position and orientation of the location corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot may be generated, and the generated result may be used as the operation for that location. In this case, the relationship between the position and orientation of the operation instruction member 5 and the position and orientation of the portion corresponding to the operation instruction member 5 in the three-dimensional model is usually determined in advance, and the operation acquisition unit 34 obtains the By using the relationship to acquire the position and orientation of the location corresponding to the operation instruction member 5 in the three-dimensional model from the position and orientation of the operation instruction member 5, the operation of that location may be acquired. Note that this case will be described later.

It should be noted that acquisition of the operation corresponding to the position of the operation instruction member 5 is performed from the time when the start instruction is received by the reception unit 33 to the time when the end instruction is received. In other words, the operation acquisition unit 34 acquires the operation according to the position of the operation instruction member 5 between the start instruction and the end instruction. The acquired operation may indicate, for example, the position of the tip of the welding torch and the orientation of the tip portion in real time.

Based on the three-dimensional model of the virtual robot stored in the storage unit 31 and the relative positional relationship acquired by the positional relationship acquisition unit 32, the image generation unit 35 generates three images at predetermined positions in the real environment. Generate a display image to display the dimensional model. Also, when an operation to the virtual robot is acquired, the image generation unit 35 generates a display image for displaying the operated three-dimensional model.

The predetermined position in the real environment where the three-dimensional model of the virtual robot is displayed may be, for example, a predetermined position in the world coordinate system. It may be a position having a predetermined positional relationship with the dimensional model. The position in the real environment where the 3D model of the virtual robot is displayed, or the predetermined positional relationship between the robot 1 and the 3D model of the virtual robot may be determined in advance, or may be determined by the operator. may be changeable. The three-dimensional model of the virtual robot may be displayed at the position of the robot 1 or may be displayed at a different position from the robot 1, for example. Displaying the three-dimensional model of the virtual robot at the position of the robot 1 means displaying the three-dimensional model so that the virtual robot is placed at the same position as the robot 1. The display may be such that the end portion on the end side (for example, the portion to be attached to the floor surface) overlaps the end portion on the base end side of the robot 1 . When the three-dimensional model of the virtual robot is displayed at a position different from that of the robot 1, the three-dimensional model may be displayed next to the robot 1, for example. In the image processing apparatus 3 according to the present embodiment, the location of the three-dimensional model of the virtual robot corresponding to the operation instruction member 5 is displayed at the position of the operation instruction member 5 moved by the operator in the actual environment. become. Therefore, when the three-dimensional model of the virtual robot is displayed at the position of the robot 1, the operation instruction member 5 may hit the arm of the robot 1 or the like. It is preferred that they are displayed in different positions. In any case, the three-dimensional model of the virtual robot will be displayed so that its position in the real environment does not change. For example, the three-dimensional model of the virtual robot is displayed so that the positional relationship with the robot 1, which is the real robot, does not change. Therefore, even if the operator changes the orientation of the display device 4, the display position of the three-dimensional model of the virtual robot in the real environment does not change.

The position of the three-dimensional model of the virtual robot in the world coordinate system is fixed. Therefore, the image generator 35 can arrange the three-dimensional model of the virtual robot in the virtual space so as to have the same positional relationship as in the real environment. For example, when the robot 1 is arranged in the real environment, the image generation unit 35 sets the arrangement position of the robot 1 so that it has a predetermined positional relationship with the arrangement position of the robot 1 in the virtual space in which the arrangement position of the robot 1 is predetermined. , a three-dimensional model of the virtual robot may be placed in the . Note that the angle of each joint of the virtual robot is the initial value when no operation is performed, and the post-operation value when the operation is performed. The angle of each joint after manipulation may be calculated, for example, by inverse kinematics using the position and orientation of the hand in the three-dimensional model of the virtual robot after manipulation, in the same way as with the real robot. The position and orientation of the hand in the three-dimensional model may be indicated by the acquisition result of the operation acquisition unit 34 . Further, the image generation unit 35 can know the relationship between the world coordinate system and the display coordinate system, which is the local coordinate system of the display device 4, from the relative positional relationship acquired by the positional relationship acquisition unit 32. The position and orientation of the display device 4 in virtual space can be specified. Therefore, the image generation unit 35 generates a two-dimensional display image for displaying the three-dimensional model by rendering the three-dimensional model of the virtual robot in the virtual space based on the position and orientation of the display device 4. be able to. When the virtual robot is manipulated, the shape of the three-dimensional model of the virtual robot in the virtual space is changed accordingly, as described above. Further, when the position and orientation of the display device 4 change in the real environment, the position and direction of the viewpoint in the virtual space are changed accordingly. By performing rendering after the change, a display image of the 3D model after the operation and a display image of the 3D model after the position and orientation of the display device 4 are changed are generated. Note that the image generator 35 generates a display image so that when the display image of the three-dimensional model of the virtual robot is displayed on the display of the display device 4, the size of the display image matches the actual environment. shall be That is, the three-dimensional model of the virtual robot displayed on the display of the display device 4 and the real robot arranged in the real environment so as to have the same relative positional relationship as the three-dimensional model are displayed via the display device 4. A display image is generated so that it has the same size when viewed from above.

Also, it may not be possible to operate the virtual robot according to the accepted operation. For example, an operation to move the hand of the virtual robot beyond its movable range or an operation to rotate it beyond its rotatable range may be accepted. In such a case, the image generator 35 does not have to generate a display image corresponding to the operation, or a display image of the three-dimensional model of the virtual robot that has moved or rotated within the possible range. may be generated. In addition, when an operation to move beyond the movable range or an operation to rotate beyond the rotatable range is accepted, for example, an alert to the effect that the accepted operation exceeds the operable range may be output to the worker.

The image output section 36 outputs the display image generated by the image generation section 35 to the display device 4 . Note that the image output unit 36 may output only the display image. In this case, on the display device 4, the display image is superimposed on the image of the real environment or the real environment itself. On the other hand, when the display device 4 has a non-transmissive display and the image processing device 3 receives an image of the real environment photographed by the display device 4, the image of the real environment and the display image are may be output to the display device 4 . This synthesis may be performed by, for example, a synthesis unit (not shown) included in the image processing device 3 . Further, by displaying the generated display image on the display device 4, for example, even if the operation instruction member 5 and the portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot are displayed so as to overlap each other. good.

When the three-dimensional model of the virtual robot is displayed at a position different from that of the real robot 1, for example, as shown in FIG. good too. A three-dimensional model 10 of the robot 1 and the virtual robot in FIG. 1 schematically shows a situation viewed through the display device 4 by a worker operating the virtual robot. In FIG. 1, for convenience of explanation, the three-dimensional model 10 of the virtual robot and the operation instruction member 5 are shown at different positions. You may display so that it may overlap.

Displaying a three-dimensional model of a virtual robot, and displaying a three-dimensional model after manipulation in accordance with manipulation of the position and posture of a predetermined portion of the three-dimensional model of the virtual robot are, for example, , is already known as shown in the above-mentioned Patent Document 1 and the like, and detailed description thereof will be omitted.

Next, the operation of the image processing device 3 will be described using the flowchart of FIG.

(Step S101) The accepting unit 33 determines whether or not an operation start instruction has been accepted. Then, when the operation start instruction is received, the process proceeds to step S102, and when not, the process proceeds to step S109.

(Step S<b>102 ) The operation acquisition unit 34 determines whether to acquire the position and orientation of the operation instruction member 5 . If the position or the like of the operation instruction member 5 is to be obtained, the process proceeds to step S103. If not, the process of step S102 is repeated until it is determined that the position or the like of the operation instruction member 5 is to be obtained. Note that the operation acquisition unit 34 may periodically determine, for example, to acquire the position and the like of the operation instruction member 5 . Further, the operation acquiring unit 34 may determine to acquire the position and the like of the operation instructing member 5 immediately after proceeding from step S101 to step S102, for example.

(Step S<b>103 ) The operation acquisition unit 34 acquires the position and orientation of the operation instruction member 5 . By periodically repeating the process of step S103, the movement of the operation instructing member 5 may be obtained as a result.

(Step S104) The operation acquisition unit 34 acquires an operation for a portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot based on the acquisition result regarding the position of the operation instruction member 5 in step S103. Acquisition of this operation may be acquisition of the position and orientation of a predetermined part of the three-dimensional model, for example. The position and orientation acquired in this way are normally the position and orientation after the operation. Note that, for example, when the position and orientation of the operation instruction member 5 are used as they are to operate a portion corresponding to the operation instruction member 5 in the three-dimensional model, the operation acquisition unit 34 acquires the operation instruction member 5 acquired in step S103. may be used as an operation for a portion corresponding to the operation instruction member 5 in the three-dimensional model.

(Step S<b>105 ) The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment and the display device 4 .

(Step S106) The image generation unit 35 uses the three-dimensional model of the virtual robot, the relative positional relationship acquired in step S105, and the operation acquired in step S104 to generate the manipulated three-dimensional model. Generate a display image.

(Step S<b>107 ) The image output unit 36 outputs the display image generated in step S<b>106 to the display device 4 . As a result, the worker can see the three-dimensional model of the virtual robot after manipulation. Normally, the location of the three-dimensional model corresponding to the operation instruction member 5 is superimposed and displayed at the position of the operation instruction member 5 that the operator is operating in the actual environment. In this way, the three-dimensional model of the virtual robot can be operated using the operation instruction member 5 existing in the real environment.

(Step S108) The accepting unit 33 determines whether or not an operation end instruction has been accepted. Then, if an instruction to end the operation is received, the process returns to step S101, and if not, the process returns to step S102.

(Step S109) The image generator 35 determines whether to generate a display image. If a display image is to be generated, the process proceeds to step S110, otherwise the process returns to step S101. Note that the image generator 35 may periodically determine, for example, to generate the display image. By making this judgment, for example, even if no operation is performed, if the position or orientation of the display device 4 is changed, the display image corresponding to the changed position or orientation is displayed on the display device 4 . will be displayed.

(Step S<b>110 ) The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment and the display device 4 .

(Step S111) The image generator 35 generates a display image of the three-dimensional model using the three-dimensional model of the virtual robot and the relative positional relationship acquired in step S110.

(Step S<b>112 ) The image output unit 36 outputs the display image generated in step S<b>111 to the display device 4 . As a result, the worker can see the three-dimensional model of the virtual robot. Then, the process returns to step S101.

The order of processing in the flowchart of FIG. 2 is an example, and the order of each step may be changed as long as the same result can be obtained. In addition, in the flowchart of FIG. 2, the processing ends when the power is turned off or an interrupt for processing end occurs.

Next, the operation of the image processing apparatus 3 according to this embodiment will be described using a specific example.

Assume that the operator wearing the display device 4 on his head displays the three-dimensional model of the virtual robot on the display (steps S109 to S112). Then, as shown in FIG. 3A, the operator manually moves the rod-shaped operation instructing member 5 to a position in the real environment where the three-dimensional model tool 10a is to be moved, and in that state, presses "start operation". Suppose you uttered Then, the voice is collected by the microphone of the reception unit 33, voice recognition is performed, and it is determined that an operation start instruction has been input (step S101). Then, using the photographed image of the operation instruction member 5 acquired by the display device 4, the operation acquisition unit 34 acquires the position and posture of the tip end side of the operation instruction member 5, which is directly used as the position of the tip of the tool 10a. And, it is assumed that an operation indicating a posture has been performed (steps S102 to S104). The operation is passed to the image generator 35 .

After that, the relative positional relationship between the real environment and the display device 4 is acquired by the positional relationship acquiring unit 32, and passed to the image generating unit 35 (step S105). Upon receiving the relative positional relationship and the operation of the tool 10a, the image generator 35 updates the position and orientation of the display device 4 in the virtual space according to the relative positional relationship, Based on the position and orientation of the tool 10a, the angle of each joint of the three-dimensional model of the virtual robot is calculated by inverse kinematics, and the angle of each joint of the three-dimensional model is changed according to the calculated joint angle. After that, the image generation unit 35 renders the three-dimensional model to generate a display image and passes it to the image output unit 36 (step S106). The display image is output to the display device 4 by the image output unit 36 (step S107). As a result, the operator can see a three-dimensional model in which the tool 10a is displayed so as to overlap the operation instructing member 5, as shown in FIG. 3B.

Assume that the operator then moves the operation instructing member 5 from the position indicated by the dashed line in FIG. 3C to the position indicated by the solid line. Accordingly, the tool 10a of the three-dimensional model of the virtual robot is also moved (steps S102 to S107). In this way, since the tool 10a of the three-dimensional model of the virtual robot follows the operation instruction member 5, the operator can easily operate the three-dimensional model of the virtual robot. In particular, by using the operation instructing member 5, the operator can easily make minute changes such as the angle. In addition, a base material to be welded is arranged in a real environment, and by moving the operation instruction member 5 so that the tip of the operation instruction member 5 is aligned with the welding line on the base material, the three-dimensional model of the virtual robot can be properly created. You will be able to operate

When teaching is performed according to this operation, the operator may appropriately input teaching instructions to the image processing apparatus 3 using a virtual input interface, other input devices, or the like. Then, correspondingly, for example, the angle of each joint at that time may be recorded as teaching data. The teaching data generated in this way may be passed to the robot control device that controls the robot 1, which is a real robot, and used for the playback operation of the robot 1. FIG. In this way, teaching data can also be generated using a three-dimensional model of the virtual robot. Assume that the operator utters "end of operation" after completing a series of operations on the three-dimensional model of the virtual robot. Then, the voice is collected by the microphone of the reception unit 33, voice recognition is performed, and it is determined that an operation end instruction has been input (step S108). Then, the series of operations is completed, and the three-dimensional model tool 10a no longer follows the operation instruction member 5. FIG.

Also, in this specific example, for convenience of explanation, the operator's hand holding the operation instructing member 5 is omitted in FIGS. 3A to 3C. Further, for example, in the case where the circumferential surface of the operation instruction member 5 is provided with patterns, protrusions, recesses, or the like for acquiring the posture of the operation instruction member 5, the photographed image of the operation instruction member 5 is used. When the position and posture of the operation instruction member 5 are acquired by using the operator, it is preferable that the operator holds the operation instruction member 5 so that the pattern of the circumferential surface of the operation instruction member 5 is included in the captured image. is.

Moreover, in the present embodiment, the case where the operation is acquired so that the position of the tip of the operation instructing member 5 is the position of the tip of the tool 10a has been mainly described, but this does not have to be the case. For example, as shown in FIG. 4, an operation may be acquired in which the tip of the tool 10a is positioned closer to the tip of the operation instructing member 5 by a distance L than the other end. In this case, for example, when the tool 10a is operated in consideration of the protrusion length of the welding wire, the pointed end of the operation instruction member 5 is moved along the weld line of the base material, and the distance L is set as the protrusion length. good. Then, the operation acquiring unit 34 may acquire an operation in which a position closer to the other end by the distance L from the tip of the operation instructing member 5 is the position of the tip of the tool 10a. By doing so, it is possible to realize an operation that considers the projection length of the three-dimensional model of the virtual robot. In this case, the acquired position of the operation instructing member 5 is different from the position of the tool 10a indicated by the acquired operation. In this way, the position indicated by the operation on the portion corresponding to the operation instruction member 5 in the three-dimensional model may be a position separated by a predetermined length from the acquired position of the operation instruction member 5 . Further, the position corresponding to the tip of the tool 10a may be indicated by a color, a ring, or the like on the rod-shaped operation instruction member 5. FIG. For example, a position closer to the other end by a distance L from the tip of the operation instructing member 5 may be painted in a color different from the others over the circumferential direction. Then, an operation in which the portion of that color becomes a predetermined position in the three-dimensional model, for example, the position of the tip of the tool 10a may be acquired. A ring-shaped member movable in the longitudinal direction may be attached to the rod-shaped operation instruction member 5 . Then, an operation in which the ring portion becomes a predetermined position in the three-dimensional model, for example, the position of the tip of the tool 10a may be acquired. In this case, the operator can arbitrarily change the position of the tip of the tool 10a, for example, by moving the position of the ring.

Moreover, in the present embodiment, the case where the tool 10a is operated by the operation instructing member 5 has been mainly described, but this does not have to be the case. A partial object such as an arm that forms the three-dimensional model 10 of the virtual robot may be operated by the operation instruction member 5 . That is, the part corresponding to the operation instructing member 5 in the three-dimensional model 10 of the virtual robot may be a predetermined arm instead of the tool, hand, or TCP. More specifically, as shown in FIG. 5, among the partial objects 10-1 to 10-8 of the three-dimensional model 10 of the virtual robot, the partial object 10-7 is operated by the operation instruction member 5. good too. That is, the position and orientation of a certain partial object 10-7 may be determined according to the position and orientation of the operation instruction member 5 in the real environment. In this case, the image generator 35 may generate a display image that can identify the partial object 10-7 operated by the operation instruction member 5, for example. More specifically, a display image may be generated in which the color and brightness of the partial object 10-7 are different from those of other partial objects. In addition, the operator may be able to select the partial object to be operated as appropriate. In this case, for example, when the operator selects a predetermined partial object in the three-dimensional model of the virtual robot displayed on the display of the display device 4 by a gesture operation such as an air tap, the selected partial object is , may become an operation target using the operation instruction member 5 .

As described above, according to the image processing apparatus 3 according to the present embodiment, by operating the operation instruction member 5 existing in the real environment, a predetermined portion of the three-dimensional model of the virtual robot in the virtual space is operated. be able to. Therefore, in the virtual space, a motion (holding motion) of virtually pinching a predetermined portion of the virtual robot with a hand is performed, and a more intuitive operation can be realized that is more realistic than moving the pinched portion. can be done. In particular, more accurate operations can be realized when performing an operation of moving along a base material existing in an actual environment, an operation of slightly changing an angle, or the like. In addition, the period during which the three-dimensional model follows the operation instruction member 5 and the period during which the three-dimensional model follows the operation instruction member 5 by acquiring the operation by the operation acquisition unit 34 between the operation start instruction and the operation end instruction accepted by the reception unit 33 and during the period until the end instruction. period, and appropriate manipulation can be achieved.

In this embodiment, the case where the operation start instruction and the operation end instruction are accepted by the accepting unit 33 has been described, but this need not be the case. For example, it may be determined that an operation start or an operation end is instructed according to the operator's operation of the operation instruction member 5 . In this case, for example, the operation acquisition unit 34 determines that the start of the operation is instructed when the rod-shaped operation instruction member 5 included in the captured image is shaken for the first time, and is shaken for the second time. At that time, it may be determined that the end of the operation has been instructed. Further, for example, the operation acquisition unit 34 determines that the start of the operation has been instructed when the operation instruction member 5 included in the captured image is superimposed on the part of the three-dimensional model corresponding to the operation instruction member 5. , it may be determined that the end of the operation has been instructed when the operation instructing member 5 is swung. Further, when the operation instruction member 5 has a light source, for example, the operation acquisition unit 34 determines that the start of the operation has been instructed when the light source of the operation instruction member 5 included in the captured image is turned on. However, it may be determined that the end of the operation has been instructed when the light is turned off. In this way, the image processing apparatus 3 does not need to have the reception unit 33 when the reception unit 33 does not receive the operation start instruction and the operation end instruction.

Moreover, in the present embodiment, the case where the rod-shaped operation instruction member 5 is used has been mainly described, but this need not be the case. For example, the operation instruction member may be a planar member. Then, for example, the tip of the tool of the three-dimensional model is positioned at a predetermined position (for example, the center, the center of gravity, or the position where a predetermined figure, convex portion, concave portion, etc. exist) on the planar member. However, an operation on the tool may be acquired such that the orientation of the tip side of the tool is the normal direction of the planar member. In this case, when the display image of the three-dimensional model is displayed on the display device 4, the portion corresponding to the operation instruction member in the three-dimensional model (for example, the tip portion of the tool) and the planar operation instruction member may be displayed so as not to overlap. For example, the tool may be displayed such that the tip of the tool contacts a predetermined position on the surface of the planar operation instruction member.

Further, in the present embodiment, the position and orientation of the operation instruction member 5 are acquired by the operation acquisition unit 34, and the operation of the location corresponding to the operation instruction member 5 of the three-dimensional model of the virtual robot is acquired according to the acquisition result. Although the explanation was mainly about the case where the Even if only the position of the operation instruction member 5 is acquired by the operation acquisition unit 34, and an operation indicating only the position of the part corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot is acquired according to the acquisition result. good. Even in this case, for example, the position of the tip of the tool, the TCP, or other parts of the three-dimensional model can be operated using the operation instruction member 5 . In this case, for the posture of the three-dimensional model of the virtual robot, for example, a predetermined posture may be used, or a posture input by another method may be used. For example, when the operator selects a predetermined portion of the three-dimensional model by air tapping or the like, the operation acquiring unit 34 may acquire an operation in which the selected position is the position of the tip of the operation instructing member 5. .

Further, in the above embodiments, each process or function may be implemented by centralized processing by a single device or single system, or may be implemented by distributed processing by multiple devices or multiple systems. It may be realized by For example, at least part of the configuration of the image processing device 3 may be physically included in a device having a display. Therefore, it may be considered that the division of devices shown in FIG. 1 is for the sake of convenience according to the functions rather than according to the physical devices.

Further, in the above-described embodiment, when the information is passed between the components, for example, when the two components that exchange the information are physically different, one of the components output of information and reception of information by the other component, or one component if the two components that pass the information are physically the same from the phase of processing corresponding to the other component to the phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component Also, information such as thresholds, formulas, addresses, etc. used by each component in processing may be stored temporarily or for a long period of time in a recording medium (not shown), even if not specified in the above description. Further, each component or an accumulation section (not shown) may accumulate information in the recording medium (not shown). Further, each component or a reading unit (not shown) may read information from the recording medium (not shown).

Further, in the above embodiment, if the information used in each component etc., for example, information such as thresholds, addresses and various set values used in processing by each component may be changed by the user, the above The user may or may not be able to change such information as appropriate, even if not explicitly stated in the description. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives a change instruction from the user and a change unit (not shown) that changes the information according to the change instruction. may The reception of the change instruction by the reception unit (not shown) may be, for example, reception from an input device, reception of information transmitted via a communication line, or reception of information read from a predetermined recording medium. .

Further, in the above embodiment, when two or more components included in the image processing apparatus 3 have communication devices, input devices, etc., the two or more components may physically have a single device. or may have separate devices.

Further, in the above embodiments, each component may be configured by dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component can be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU. During the execution, the program execution unit may execute the program while accessing the storage unit or recording medium. Further, the program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). good. Also, this program may be used as a program constituting a program product. Also, the number of computers that execute the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible and are also included within the scope of the present invention.

3 image processing device 4 display device 5 operation instruction member 31 storage unit 32 positional relationship acquisition unit 33 reception unit 34 operation acquisition unit 35 image generation unit 36 image output unit

Claims (3)

a storage unit storing a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment;
an operation acquisition unit that acquires a position of an operation instruction member existing in a real environment and acquires an operation for a location corresponding to the operation instruction member in the three-dimensional model of the virtual robot according to the position;
a positional relationship acquiring unit that acquires a relative positional relationship between the real environment in which the real robot exists and a display device that displays an image superimposed on the image of the real environment or the real environment itself;
an image generation unit that generates a display image for displaying the manipulated three-dimensional model at a predetermined position in a real environment based on the three-dimensional model and the relative positional relationship;
and an image output unit that outputs the display image to a display device. The operation instruction member is a rod-shaped member,
2. The image processing apparatus according to claim 1, wherein said operation acquisition unit acquires an operation for a tool of said three-dimensional model according to a position of said operation instruction member. further comprising a reception unit that receives a start instruction for instructing the start time of the operation using the operation instruction member and an end instruction for instructing the end time of the operation,
3. The image processing apparatus according to claim 1, wherein said operation acquisition unit acquires an operation corresponding to a position of said operation instruction member between said start instruction and said end instruction.

Images