JP6643000B2 - Virtual environment creation method, robot apparatus control method, robot system, and information processing apparatus - Google Patents

Description

The present invention relates to a virtual environment creation method for generating a 3D model of an object installed in an installation environment of a production apparatus and arranging the 3D model in a 3D virtual environment, a control method of a robot apparatus , a robot system , and an information processing apparatus .

  2. Description of the Related Art In recent years, in the field of industrial production, a production (manufacturing) apparatus using a multi-joint robot apparatus (hereinafter, referred to as a robot apparatus) capable of realizing a complex and high-speed article manufacturing operation like a human hand is becoming popular. When assembling with this type of robot device, it is necessary to create a robot operation program. At this time, an operation program may be created using a 3D virtual environment using computer graphics.

  Such a 3D virtual environment for robot programming (teaching) is constructed by arranging a virtual 3D model such as a robot device or a work (work) created by a CAD (Computer aided design) tool. Such a 3D virtual environment can be displayed on a display, and an operation program can be created while confirming the operation on, for example, a GUI (Graphical user interface). A robot programming environment configured to operate a 3D virtual environment with a GUI has an advantage that intuitive operation is possible.

  When constructing the 3D virtual environment as described above, usually, design information (size, shape, geometry) of the robot device and the work (work) is often known in advance. Therefore, a 3D model constructed by CAD processing or the like based on design information can often be prepared as spatial information to be arranged in a virtual environment.

  However, spatial information created on CAD using only design information may be slightly different in size and installation position from objects in the real environment. This is because an object (a robot device or a work) in the real environment has a tolerance or a part manufacturing error.

  For this reason, conventionally, an operation program for a robot device created in a virtual environment is adjusted through an operation check in a real environment before being used in a real environment. For example, a confirmation operation is performed one step at a time using a robot device installed in a real environment or an object to be worked. Then, for example, a portion that does not match the real environment is corrected using an operation terminal such as a teaching pendant. However, in recent years, assembling processes using a robot apparatus have become more complicated, and the number of man-hours and time required for adjustment work performed in such a real environment tend to be enormous.

  In view of the circumstances described above, a technique has been proposed in which an image of a real environment is captured using an imaging device such as a camera, and the captured image is reflected on a virtual environment. For example, a technology has been proposed in which an imaging device is provided at the tip of a robot device to acquire spatial information in a real environment by imaging a workpiece, and to convert the acquired spatial information of a work object to a virtual environment (for example, Patent Document 1 below). In addition, a technique has been proposed in which spatial information obtained by an imaging device such as a CT scanner is compared with spatial information obtained from design information to perform dimension correction and the like (for example, Patent Document 2 below).

JP-A-5-245793 JP-A-2004-271222

  However, in Patent Literature 1, the imaging target is limited to the work, and there is no operation of capturing an image of an object other than the work and acquiring spatial information. Further, there is no mention of the case where there is spatial information created from design information in advance by CAD or the like or the handling of such spatial information. Further, in Patent Document 2, although the spatial information created in advance and the spatial information obtained by imaging are compared, if there is a difference between the objects present in both spatial information, or if the object is only one of the two spatial information, There is no mention of the handling of such cases.

  Actually, it is not rare that there is a difference between spatial information obtained by environment recognition using an imaging device and the like and spatial information created in advance. For example, it is difficult to determine in advance how a flexible object such as a cable is arranged in a real environment on a design information or a 3D model CAD-processed based on the design information.

  For example, it is not uncommon for a flexible object such as a cable to be placed in an installation environment of a robot device. For example, a robot device and a control device may be connected, or workpieces (workpieces) may be connected by a cable. In such a case, it is necessary to create an operation program of the robot device so that the robot arm and the end effector at the tip thereof do not buffer the cable. However, conventionally, in the case of a flexible object such as a cable, a space larger than the actual occupied area is registered on the virtual environment as a placement area, and the handling is performed as a robot intrusion prohibited area. . For this reason, the robot device may not always be able to operate efficiently, and there is a problem that the tact time of the actual process increases.

  Further, at the installation site of the robot device, there is a possibility that an object such as a cable may be added as necessary or the handling may be changed. For this reason, when the installation environment of the robot apparatus is imaged by a 3D imaging device or the like, an object such as a cable may be in a state where the corresponding spatial information does not exist in the virtual environment created from the design information in advance. is there. In such a case, there is a problem that it is difficult to automatically determine whether or not to reflect the spatial information of the imaged object in the virtual environment.

  An object of the present invention is to enable a state close to a real environment to be appropriately reproduced on a 3D virtual environment from spatial information (3D data) obtained by a 3D data acquisition device.

In order to solve the above problems, in the present invention, the control device generates a 3D model of an object installed in the installation environment by using 3D data acquired by the 3D data acquisition device from the installation environment of the production device, In a virtual environment creation method for arranging in a 3D virtual environment corresponding to an installation environment of a device,
A first matching step in which the control device matches 3D data acquired by the 3D data acquisition device with first dictionary data including shape information of an object placed in the installation environment; A second collation step of collating 3D data acquired by the 3D data acquisition device with second dictionary data including information on a flexible object connected to an object placed in the installation environment; controller, based on the first or second verification process, look including an output processing step, which generates a 3D model of the object being disposed in the installation environment to place the 3D virtual environment, said first The second dictionary data includes connection characteristic information including at least one of a position and a direction in which the flexible object is connected to the object arranged in the installation environment, and the control device includes the second illumination device. In step, characterized the 3D data acquired by the 3D data acquisition device, a configuration for matching, and the connection characteristic information.

  According to the present invention, spatial information (3D data) obtained by a 3D data acquisition device from an installation environment of a production device, first dictionary data generated in advance from shape information of an object placed in the installation environment, Collate. In addition, collation is performed with second dictionary data generated in advance from feature information of an object that may be placed in the installation environment. Then, a 3D model of the object corresponding to the first or second dictionary data in the 3D data is generated and arranged in the 3D virtual environment. For example, an object placed in the installation environment, such as a production device (robot device or the like) itself or a work (work), is identified by collating with the first dictionary data. Also, for example, an object such as a cable that may be deformed and may be placed in the installation environment is identified by collating with the second dictionary data. Therefore, an object that may be deformed, such as a cable, for which shape data cannot be prepared in the first dictionary data can be identified by collation with the second dictionary data. For this reason, using the spatial information (3D data) obtained by the 3D data acquisition device from the real environment, the 3D data of each object existing in the real environment can be appropriately reproduced on the 3D virtual environment.

FIG. 1 is an explanatory diagram illustrating a schematic configuration of an entire robot system according to a first embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating a schematic configuration of a robot device. FIG. 2 is a block diagram illustrating a functional configuration of a robot control device. It is explanatory drawing of an environment recognition apparatus. It is an explanatory view of a robot operation program creation device. It is a schematic diagram which shows the real environment of an assembly environment. FIG. 4 is a schematic diagram showing space information created in advance. It is a schematic diagram which shows the space information which recognized environment. (A) is a schematic view of the previously created spatial information viewed from above, and (b) is a schematic view showing a connection list. FIG. 4 is a flowchart illustrating a flow of an environment recognition process according to the first embodiment of the present invention. (A) is a schematic diagram of an environment recognition result input process, (b) is a schematic diagram after a planar object extraction process, and (c) is a schematic diagram after an object separation and collation process. (A) is a schematic diagram showing the object separation and matching process, and (b) is a schematic diagram showing the matching of the object separation and matching process. (A) is a schematic diagram in which an object group is arranged on a grid at the time of object separation and matching processing, and (b) is a schematic diagram in which a region where an object may exist is additionally shown in (a). (A) is a schematic diagram illustrating a virtual obstacle set in a conventional virtual environment, (b) is a schematic diagram illustrating a virtual obstacle in the first embodiment, and (c) is an environment recognition of the virtual obstacle. It is a schematic diagram when a result is deformed. FIG. 5 is an explanatory diagram illustrating an effect of environment recognition according to the first exemplary embodiment. Regarding Embodiment 2 of the present invention, (a) and (b) are schematic diagrams illustrating deformation of a connected object caused by a change in the distance of a work, and (c) is a schematic diagram illustrating integration of an entry prohibited area. . It is a schematic diagram which shows the space information which recognized the environment concerning Example 3 of this invention. FIG. 14 is a flowchart illustrating a flow of an environment recognition process according to the third embodiment. It is a schematic diagram which shows the space information which recognized the environment concerning Example 4 of this invention. FIG. 14 is a flowchart illustrating a flow of an environment recognition process according to a fourth embodiment. It is a schematic diagram which shows the space information which recognized the environment concerning Example 5 of this invention. (A) according to the sixth embodiment of the present invention is a schematic diagram showing the space information in which the environment is recognized in a bird's-eye view state, and (b) is a schematic diagram showing an expanded connection list.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to embodiments shown in the accompanying drawings. It should be noted that the following embodiments are merely examples, and for example, those skilled in the art can appropriately change the detailed configuration without departing from the spirit of the present invention. Numerical values taken in the following description are reference numerical values, and do not limit the present invention.

  FIG. 1 shows a schematic functional configuration of the entire robot system to which the present invention can be applied. In FIG. 1, an environment recognition device 500 is configured to be able to execute the virtual environment creation method of the present embodiment. The environment recognition device 500 inputs space information 520 created in advance from the recording disk 522. This space information 520 corresponds to space information 700 (first dictionary data: FIG. 7) described later. The environment recognition device 500 creates virtual environment space information 530 (3D virtual environment) that can be used by the robot operation program creation device 550 by a method described later, and outputs it to the recording disk 532.

  The robot operation program creation device 550 receives the virtual environment space information 530 (3D virtual environment) created by the environment recognition device 500, creates a robot operation program 570, and outputs it to the recording disk 580. The robot device 100 inputs the robot operation program 570 of the recording disk 580 and operates the robot arm.

  FIG. 2 illustrates a schematic configuration of the robot apparatus according to the first embodiment of the present invention. The robot apparatus 100 of FIG. 2 includes, for example, a robot arm 200 having a multi-joint configuration, and a robot controller 300 that controls the robot arm 200.

  In addition, the robot device 100 includes a teaching pendant 400 as a teaching device that transmits data of a plurality of teaching points to the robot control device 300. The teaching pendant 400 is operated by an operator and is used to specify the operation of the robot arm 200 and the robot control device 300.

  In the first embodiment, the robot arm 200 is a robot arm having six joints (six axes). The robot arm 200 has a plurality (six) of actuators 201 to 206 that rotate the joints J1 to J6 around the joint axes A1 to A6, respectively.

  The robot arm 200 can direct its hand (the tip of the robot arm 200) to a posture in any three directions at any three-dimensional position within the movable range. In general, the position and orientation of the robot arm 200 can be represented by a three-dimensional (XYZ) coordinate system.

  Reference symbol To in FIG. 2 denotes a three-dimensional coordinate system arranged with the base of the robot arm 200 as a reference (origin), and Te denotes a coordinate system arranged with a predetermined part of the hand of the robot arm 200 as a reference (origin). 3 shows a dimensional coordinate system.

  In the first embodiment, the actuators 201 to 206 include the electric motors 211 to 216 and the reduction gears 221 to 226 connected to the electric motors 211 to 216. Note that the configuration of the actuators 201 to 206 is not limited to the above configuration, and may be, for example, an artificial muscle or the like.

  In the first embodiment, a case where each of the joints J1 to J6 is a rotating joint will be described. In this case, the “position of the joint” means a rotation angle of the joint axis from a reference angle defined for the joint. Note that each joint may be a linear joint. In this case, the “joint position” is position information of a linear joint from a reference position defined for the joint.

  The robot arm 200 further has a servo control device 230 as a drive control unit that drives and controls the electric motors 211 to 216 of the actuators 201 to 206.

  The servo controller 230 outputs a current to each of the electric motors 211 to 216 based on the input position command (target position) so that the positions of the joints J1 to J6 follow the position command. Control the operation of.

  The robot control device 300 receives an input of a plurality of teaching points (teaching point sequence) from the teaching pendant 400.

  The robot controller 300 generates a position command to be output to the servo controller 230 at predetermined time intervals based on the teaching points so that the joints J1 to J6 of the robot arm 200 operate by sequentially following a plurality of teaching points. And outputs a position command at predetermined time intervals.

  The teaching point is a vector (teaching point vector) including the teaching positions of the joints (six joints) J1 to J6 as vector components. The position command finally obtained by the robot control device 300 is a vector including the target positions of the joints (six joints) J1 to J6 as vector components. That is, the robot controller 300 calculates the trajectory from the input teaching points, generates a number of position commands along the trajectory, and outputs these position commands to the servo controller 230 at predetermined time intervals.

  Here, the following two types are mainly considered as control methods for controlling the position and orientation of the robot arm 200 by teaching.

(1) Motion in configuration space
A coordinate system that uses the joint angle of the robot as a displacement is called a joint space. This joint space moving method is a method in which a teaching point of the robot arm 200 is designated in the joint space, and the robot is moved along a trajectory determined by the designated teaching point.

(2) Motion in task space
A coordinate system represented by a three-dimensional position and orientation is called a task space. This task space moving method is a method in which a teaching point is designated in the task space, and the robot arm 200 is moved along a trajectory determined by the designated teaching point and the hand of the robot arm 200.

  FIG. 2 schematically illustrates four teaching points p1, p2, p3, and p4. In any case, the robot arm 200 is controlled to move along a trajectory determined by the teaching point. The arrows shown together with the teaching points p1, p2, p3, and p4 correspond to the above-described three-dimensional coordinate system Te arranged at the tip of the robot arm 200.

  The data representation of the teaching points p1, p2, p3, and p4 in FIG. 2 is not necessarily coordinate information in a three-dimensional space. For example, in the case of the above joint space movement (1), the teaching point is represented by the joint angles of all the joints J1 to J6 (the rotation angles of the respective motors).

  On the other hand, in the case of the task space movement (2) described above, the teaching point corresponds to the position and posture of the hand part of the robot arm 200 in the three-dimensional space, which is represented by p1, p2, p3, as shown in FIG. p4 is represented as a coordinate system.

  In each embodiment of this specification, any of the above-described position and orientation control methods (1) and (2) may be used in robot programming (teaching).

  FIG. 3 shows a configuration of the robot control device 300. In the first embodiment, as shown in FIG. 3, the robot control device 300 is configured as a computer device including a CPU 301 as a calculation unit.

  3, the robot controller 300 includes a ROM 302, a RAM 303, and an HDD 304 as storage units. In addition, the robot controller 300 includes a recording disk drive f and various interfaces 306 to 309.

  3, a ROM 301, a RAM 303, an HDD 304, a recording disk drive 305, and various interfaces 306 to 309 are connected to a CPU 301 via a bus 310. The ROM 302 stores a basic program such as a BIOS.

  The RAM 303 is a storage unit that temporarily stores the calculation processing result of the CPU 301. The HDD 304 is a storage unit for storing various types of data that are the results of arithmetic processing, and stores a program 330.

  The program 330 is a program for causing the CPU 301 to execute various arithmetic processes. The CPU 301 executes various arithmetic processes based on the program 330 stored in the HDD 304.

  The teaching pendant 400 is connected to the interface 306, and the CPU 301 receives, via the interface 306 and the bus 310, the input of the teaching point data and the designation of the interpolation method from the teaching pendant 400.

  In the HDD 304, data (data of a position command) indicating a calculation result by the CPU 301 and the like are stored under a command of the CPU 301.

  The servo controller 230 of the robot arm 200 is connected to the interface 309, and the CPU 301 outputs position command data to the servo controller 230 via the bus 310 and the interface 309 at predetermined time intervals.

  The monitor 321 is connected to the interface 307 (not shown in FIG. 2). The monitor 321 is used for displaying various images and text data expressing the current state of the robot arm 200 and the like.

  The interface 308 is configured so that an external storage device 322 such as a rewritable nonvolatile memory or an external HDD can be connected. The recording disk drive 305 can read or write various data and programs on the recording disk 580.

  FIG. 4 is a schematic diagram of the environment recognition device 500 according to the first embodiment. The environment recognition device 500 includes an environment input device 501 and an environment recognition processing device 502, and inputs, from a recording disk 522, space information 520 created in advance based on design information of the robot arm 200 and a work.

  The environment input device 501 includes a 3D data acquisition device such as a 3D camera for acquiring three-dimensional spatial information of an installation environment of the robot arm 200. The shooting method of the 3D camera is not limited to the stereo shooting method, and may be configured by an arbitrary shooting method. Further, if possible mechanically, it is also conceivable to use a shape measuring device in which a large number of probes (contact probes) are arranged. The environment input device 501 is connected to the environment recognition processing device 502 via various parallel (or serial) interfaces. The environment input device 501 inputs the three-dimensional space information acquired from the installation environment of the robot arm 200 to the environment recognition processing device 502.

  The environment recognition processing device 502 converts the space information 520 created based on the design information and the three-dimensional space information acquired from the installation environment of the robot arm 200 by the environment input device 501 into a virtual environment for the virtual environment using the environment recognition processing device 502. The spatial information 530 is created. The virtual environment space information 530 can be output to, for example, a recording disk 532.

  The environment recognition processing device 502 is a computer that processes spatial information obtained from the environment input device 501, and includes a monitor, a keyboard, and a recording drive. The environment recognition processing device 502 is a computer device using hardware of, for example, an off-the-shelf PC (Personal Computer) (similar to the robot control device 300), as shown in FIG. is there.

  The specific configuration of the environment recognition processing device 502 is substantially the same as the configuration of FIG. 3 except for blocks related to the robot arm, for example, except for the teaching pendant 400, the servo control device 230, the electric motors 211 to 216, the interfaces 306, 309, and the like. Are equivalent. Hereinafter, the CPU 301, the ROM 302, the RAM 303, the HDD 304, and the like in FIG. 3 may be referred to as (equivalent) components of the environment recognition processing device 502.

  FIG. 4 shows an environment recognition unit 510 as a function of the environment recognition processing device 502. The environment recognition unit 510 is implemented by hardware of a computer (CPU 301 in FIG. 3) and its software. In particular, a virtual environment creation program corresponding to a control procedure executed by a computer (CPU 301) described later is stored in, for example, the ROM 302 or the HDD 304 in FIG. These storage means such as the ROM 302 and the HDD 304 are, of course, computer-readable recording media. These (parts of) a computer-readable recording medium such as the ROM 302 or the HDD 304 may be constituted by a removable flash memory device or a magnetic / optical disk. Further, a configuration in which the software of the environment recognition processing device 502 is downloaded via a network or the like, and is introduced into a recording medium such as the HDD 304, or updates the installed software is also conceivable. The storage destination of the software of the robot operation program creation device 550 and the software (robot control program) of the robot control device 300, and the configuration for introduction and update are the same as described above.

  The environment recognition unit 510 includes functional blocks of an environment recognition information input unit 511, a plane extraction unit 512, an object separation unit 513, a collation unit 514, and a space information creation unit 515.

  The space information 520 created in advance is space information created by CAD or the like from design information of the robot arm 200 and a work operated by the robot arm 200, and is input to the environment recognition processing device 502 via the recording disk 522, for example.

  The environment recognition unit 510 of the environment recognition processing device 502 performs virtual environment space information based on three-dimensional space information obtained by the environment input device 501 and space information 520 created from design information in advance by a recognition process described later. 530 is created. The created virtual environment space information 530 is stored on the recording disk 532.

  The environment recognition unit 510 of the environment recognition processing device 502 includes, for example, a three-dimensional space information obtained from the environment in which the robot is installed by the environment input device 501 and a space that is a 3D model corresponding to the robot arm 200 or the work created in advance from the design information. The information 520 is collated. Then, spatial information corresponding to the spatial information 520 of the object in the three-dimensional spatial information obtained by the environment input device 501 is arranged (registered) in the virtual environment spatial information 530.

  The spatial information 520 corresponding to the robot arm 200 and the work created in advance from the design information may be considered as first dictionary data used by the environment recognition unit 510 for environment recognition.

  However, in the three-dimensional spatial information obtained by the environment input device 501 from the actual robot installation environment, even if the three-dimensional spatial information is compared with the spatial information 520 (first dictionary data) created from the design information in advance, an object that does not correspond to the spatial information 520 is obtained. May contain spatial information. In addition, the spatial information 520 (first dictionary data) created in advance from the design information may include an essential object to be arranged in an actual robot installation environment. Then, for example, there is a possibility that spatial information of an indispensable pertinent object to be arranged in the actual robot installation environment is not included in the three-dimensional spatial information obtained by the environment input device 501 due to, for example, forgetting to arrange.

  In particular, it is difficult to determine in advance how to place a flexible object such as a cable in a real environment on a design information or a 3D model CAD-processed based on the design information.

  In the first embodiment, in order to cope with a case where an object (for example, a work: an object to be worked) connected via a flexible object such as a cable is arranged, the space information 520 (the Second dictionary data different from the first dictionary data) is used.

  FIG. 5 shows a schematic configuration of the robot operation program creation device 550 of the first embodiment.

  The robot operation program creation device 550 is a computer that inputs the virtual environment space information 530 stored on the recording disk 532, and includes a monitor, a keyboard, and a recording drive. Like the environment recognition processing device 502, the robot operation program creating device 550 is also a computer device using, for example, hardware of an existing PC (Personal Computer).

  The specific configuration of the robot operation program creation device 550 is substantially the same as, for example, the configuration of FIG. However, in the case of the robot operation program creation device 550, blocks related to the robot arm, for example, the teaching pendant 400, the servo control device 230, the electric motors 211 to 216, the interfaces 306 and 309 are not required. Hereinafter, the components of the robot operation program creation device 550 may also refer to the CPU 301, the ROM 302, the RAM 303, the HDD 304, and the like in FIG.

  Note that the robot operation program creation device 550 and the environment recognition processing device 502 can also be implemented by installing software for each on the same PC hardware. In this case, the virtual environment space information 530 may be transmitted and received on the same PC hardware, and the virtual environment space information 530 may be transmitted and received via a memory or HDD on a computer without using the recording disk 532. Good.

  The functional blocks of the robot operation program creation device 550 include a robot operation program creation simulator 560 as shown on the right side of FIG. The robot operation program creation simulator 560 includes a virtual environment display unit 561 and a robot operation creation unit 562. These functional blocks (560: 561, 562) are implemented by hardware of a computer (CPU 301 in FIG. 3) and its software (prepared and stored in ROM 302 or HDD 304 in FIG. 3).

  Specifically, the robot operation program creating simulator 560 operates on a GUI constructed on a monitor of a PC. The GUI displays a virtual environment representing the robot arm 200 and an object in the installation environment using the virtual environment space information 530 received from the environment recognition processing device 502 (virtual environment display unit 561). Further, the robot operation creation unit 562 is configured to be able to operate a virtual representation of an object displayed in a virtual environment (virtual environment display unit 561) via user interface means such as a pointing device and a keyboard. As the pointing device of the user interface, for example, a mouse, a trackpad, a trackball, or the like can be used, and a touch panel disposed on a monitor can be used. Through such a user operation, for example, an operation program for operating the robot arm 200 to take a desired position and orientation can be created (the robot operation creation unit 562).

  The robot operation program 570 created by the user on the robot operation program creation device 550 is sent to the robot control device 300, and the robot arm 200 can be operated according to the robot operation program 570. The robot operation program 570 can be stored in, for example, a recording disk 580 and sent to the robot controller 300 via the recording disk drive 305 of the robot controller 300. Note that, in order to transfer the robot operation program 570 from the robot operation program creation device 550 to the robot control device 300, a network communication may be used in addition to using the recording disk drive 305.

  Hereinafter, the first embodiment will be described while more specifically showing the robot arm 200 and its installation environment.

  FIG. 6 shows the robot arm 200, the work 610, and the tools 611 to 612 arranged in the actual installation environment 600. The work 610 is, for example, a fixture used by the robot arm 200 for assembly, and the jigs 611 to 612 are tools used by the robot arm 200 for assembly. In FIG. 6, connection objects 620 to 621 are cables, and are connected to the work objects 610 to 612 and the robot arm 200. The installation surface 650 is a plane on which objects necessary for assembly, such as the work objects 610 to 612 and the robot arm 200, are installed.

  On the other hand, FIG. 7 schematically shows space information 700 (first dictionary data) created in advance. This spatial information 700 (first dictionary data) is created in advance in the form of 3D data by a CAD tool or the like, based on design information of an object placed in the robot installation environment. The space information 700 (first dictionary data) in FIG. 7 corresponds to the space information 520 in FIGS. 1 and 4, and is input to the environment recognition device 500 via the recording disk 522 or the like.

  In FIG. 7, a robot arm 701 is spatial information (3D model) corresponding to the robot arm 200 in the real environment, and work objects 710 to 712 are spatial information corresponding to the work objects 610 to 612 in the real environment. (3D model). The plane 750 is space information (3D model) corresponding to the installation surface 650 in the real environment. As for such spatial information (3D model), there is numerical design information on the shape and dimensions of each part, and it can be automatically generated in advance using a CAD tool or the like.

  Unlike the real environment in FIG. 6, it is difficult to determine in advance how to arrange the connectors 620 to 621 such as cables in the spatial information 700 in FIG. 7. not exist.

  FIG. 8 schematically shows space information 800 whose environment has been recognized by the environment recognition device 500. FIG. 8 shows 3D data acquired from the installation environment 600 (real environment) in FIG. 6 by the environment input device 501, the space information 700 in FIG. 7 and a connection list 900 (second dictionary data) described later. The result of recognition of the spatial environment by the environment recognition processing device 502 is shown.

  8, work objects 810 to 812 are the environment recognition results of the work objects 610 to 612 in the real environment in FIG. The robot arm 801 is the environment recognition result of the robot arm 200 in the real environment in FIG. 6, the connected objects 820 to 821 correspond to the environment recognition result of the connected objects 620 to 621 in the real environment, and the plane 850 corresponds to the environment recognition result of the installation surface 650 of the real environment.

  Hereinafter, information processing for acquiring the virtual environment (spatial information 800) generated as shown in FIG. 8 will be described in detail with reference to FIGS.

  FIG. 9A shows the space information 700 obtained from the design information in FIG. In FIG. 9A, height information is omitted for simplification, and only the XY plane is shown. 7, objects A, B (901, 902) and C (903) correspond to the work objects 710 to 712 in FIG. 7, and D (904) corresponds to the robot arm 701. .., A, B, C, D,... Are the IDs of the objects for convenience, and in this embodiment, are mainly used to identify each object in the spatial information.

  In FIG. 9A, curves (broken lines) connecting the objects A and B and C and D respectively conceptually represent connections between the objects by cables (620 and 621 in FIG. 6). In this example, the cables (620 and 621 in FIG. 6) are actually unknown connections, and are not directly included in the space information 700 obtained from the design information in FIG.

  However, in the first embodiment, the work 610 and the jigs 611 to 612 (FIG. 6) arranged on the installation plane (650) are connected to the cable (620 in FIG. 6) as shown in FIGS. , 621) are known in advance (highly likely). That is, it has been found in advance that objects A, B, C, and D that are "connected" by a flexible object such as a cable are used.

  In such a case, by preparing a connection list 900 (second dictionary data) as shown in FIG. 9B, the environment information processing device 502 can reliably and rapidly perform spatial information as shown in FIG. 800 (environment recognition result) can be generated.

  A connection list 900 (second dictionary data) in FIG. 9B is a list of feature information of objects A, B, C, and D that are “connected” by a flexible object such as a cable. The connection list 900 (second dictionary data) is arranged, for example, in a memory area (for example, the ROM 302 or the RAM 303) used by the environment recognition processing device 502. Alternatively, the connection list 900 may be stored in the HDD 304 or the like before being expanded in the memory area (the ROM 302 or the RAM 303). The recording format of the connection list 900 (second dictionary data) is arbitrary. For example, a storage format such as a table memory or an array can be used in the memory area. Further, on the HDD 304, the connection list 900 (second dictionary data) may be stored using an arbitrary file format in a format that can be expanded into a format such as a table memory or an array.

  In the case of an object (flexible object) such as a cable, the shape and dimensional information known in advance is often at most about the diameter or the entire length. In addition, a flexible object such as a cable cannot be predicted in advance when it is arranged.

  For this reason, the connection list 900 (second dictionary data) of the present embodiment includes the position and the direction of the occurrence of the “connection” of the objects A, B, C, and D “connected” by a flexible object such as a cable. Such characteristic information is listed (table memory).

  That is, when a cable is connected to the robot arm 200 (701) or the work, the position of the connector (or the through hole from which the cable is directly led) for the connection is usually known. Alternatively, it is possible to design the connection position of such a cable.

  In the case of the objects A and B in FIG. 9A, the cable comes out of the object A right, and the cable comes out of the object B left. In the case of the objects C and D, the cable comes out from the object (top), and the cable comes out from the object B (bottom). Note that the connection directions of right (right), left (left), top (top), and bottom (bottom) are merely display selected for convenience on the planar display of FIG. The expression format of the dictionary data) on the memory is arbitrary. Here, for simplicity, classification is performed only in four directions. However, in the connection list 900 (second dictionary data), an expression format that can be further subdivided to express directions is prepared. Can be.

  In FIG. 9A, the connection position of the cable divides a plane corresponding to the installation surface 650 into an a × b grid. In the connection list 900 (second dictionary data: FIG. 9B) of the present embodiment, the objects A to D are mapped on this grid, and the cable connection position of each object is defined by the position. I have. Here, a and b are arbitrary integers, and a 6 × 4 grid is used in FIG. 9A.

  In this way, by treating the space for which space recognition is performed by partitioning (coarsely) with the grid, (part of) the calculation by the CPU (301) is simplified, and the processing load can be reduced.

  In this embodiment, a connection list 900 (second dictionary data: FIG. 9B) configured as described above is prepared. Then, the collating unit 514 (FIG. 4) of the environment recognition processing device 502 uses the environment input device 501 to obtain the space information obtained from the installation environment 600 as shown in FIG. 6 and the connection list 900 (second dictionary data: FIG. b)) collate.

  Then, as a result of the matching by the matching unit 514, by arranging the spatial information (3D model) of the corresponding object, an environment recognition result such as the spatial information 800 in FIG. 8 can be obtained. At this time, the environment recognition device 500 generates, for example, spatial information (3D model) of the object determined to correspond to the connection list 900 by the matching unit 514. Then, by combining with (or adding to) the spatial information 700 created in advance based on the design information or the like in FIG. 7, an environment recognition result like the spatial information 800 in FIG. 8 can be obtained.

  According to the connection list 900 (second dictionary data: FIG. 9B) of the present embodiment, the position and shape of the spatial information recognized as the real environment due to factors such as tolerances and component manufacturing errors, such as the above-described cable, are strict. Are configured using the feature information of an object (connection medium) that does not match the. For example, in the case of the object such as the above-mentioned cable, the characteristic information is the connection direction and the connection position as described above. Further, since the characteristic information of the connection list 900 (second dictionary data: FIG. 9B) uses expressions such as connection positions on the grid division of the installation space and connection directions, the position at the time of installation is used. The matching in the matching unit 514 can be performed by absorbing errors in the shape and shape.

  As described above, in the example of the characteristic information in FIGS. 9A and 9B, height information is omitted for simplification, but the connection list 900 (second dictionary data: FIG. In)), the connection position can be defined using an a × b × c three-dimensional grid in consideration of height information.

  As shown in FIG. 9B, in the connection list 900, features such as connection position information (905) and connection direction information (906) are associated with each object ID (A, B, C, D). Information is recorded. These pieces of feature information are (905, 906), for example, regarding an object that is arranged in a robot installation environment and may be connected to each other via a connection object, a connection position between the object and the connection medium, and The connection characteristic information represents the connection direction.

  The feature information (especially connection characteristic information for an object connected by a connection object) in the connection list 900 (second dictionary data) in the present embodiment is the same as the space information 700 (first dictionary data) created from the design information. Thus, it does not include shape information such as 3D data.

  In this regard, when the characteristic information or the connection characteristic information is collated with the space information obtained from the installation environment 600 by the environment input device 501, the load of the operation of the CPU (301) is small. As is well known, in the case of collation between spatial information (3D data) obtained from the real environment and spatial information such as 3D data (for example, CAD data) of the spatial information 700 (first dictionary data), complicated processing is required. In some cases, a matching process must be performed. However, the characteristic information of the connection list 900 (second dictionary data) of the present embodiment is, for example, connection characteristic information described by a connection position and a connection direction, and includes spatial information (3D data) obtained from a real environment. Can be executed with a small operation cost. In addition, the feature information of the connection list 900 (second dictionary data) of the present embodiment is configured with gradual characteristic definitions such as connection positions and connection directions. For this reason, when collating with a flexible object such as a cable, the object can be reliably captured even if there is a deformation.

  The connection position information (905) of the connection list 900 in FIG. 9B includes a position X and a position Y, which are represented by grid numbers in FIG. 9A. Although the position in the height direction (Z) is omitted in FIG. 9B for simplification as described above, the information in the height direction (Z) is recorded in the connection position information (905). For collation, for example, a field at position Z is added. The information in the height direction (Z) may also be represented by cell numbers roughly dividing the space represented by the Z coordinate by a grid.

  The information (906) regarding the connection direction of the connection list 900 is expressed by mnemonics such as right (right), left (left), top (bottom), and bottom (bottom) as described above. As the information (906) regarding the connection direction, an arbitrary mnemonic such as east (E), west (W), south (N), and north (N) may be used.

  FIG. 10 shows an overall flow of a process of teaching (programming) the operation of the robot device 100 using information processing relating to the virtual environment in the present embodiment. The control procedure of FIG. 10 is executed by the environment recognition device 500 and the robot operation program creation device 550. In this processing, using the virtual environment created by the environment recognition device 500, the robot operation can be taught via the robot operation program creation device 550.

  In particular, in the virtual environment recognition process performed by the environment recognition device 500, the space information 700 (FIG. 7: first dictionary data) created in advance from the design information and the connection list 900 (FIGS. 9A and 9B: 2 dictionary data).

  In step S901 in FIG. 10, an environment recognition result input process is performed. Here, the space information obtained from the installation environment 600 (FIG. 6) of the robot apparatus 100 via the environment input device 501 is input to the environment recognition information input unit 511. If the environment input device 501 is a device that acquires a 3D image such as a 3D (stereo) camera, the spatial information is in the form of 3D image data.

  Here, in the installation environment 600 of the robot apparatus 100, as shown in FIG. 6, in addition to the robot arm 200, a work 610 and jigs 611 to 612 (FIG. 6) are arranged. The work 610 and the jig / tool 611, and the robot arm 200 and the jig / tool 612 are connected by cables (620 and 621).

  In step S902, a planar object extraction process is performed on the spatial information input from the environment input device 501. This on-plane object extraction processing is performed by the plane extraction unit 512. This process is for extracting an object existing on the installation plane of the robot device 100 from the spatial information input from the environment input device 501. This processing is an object extraction processing, but may be a plane removal processing on one surface.

  Next, object separation and collation processing are performed in step S903. The object separation processing is performed by the object separation unit 513. This object separation processing is performed by a known method such as detection of a contour of spatial information input from the environment input device 501 and analysis of continuity of pixel values. At the stage where the object separation processing is completed, it is not yet identified whether the separated object is a robot device or a work, for example. However, in step S903, matching with the spatial information 700 (FIG. 7: first dictionary data) created in advance from the design information is performed. Thus, an object included in the space information 700 (FIG. 7: first dictionary data) is identified in the space information acquired from the environment input device 501.

  In the example of FIG. 7, a robot arm 701, a work 710, and jigs 711 and 712 are included. Is determined by the collation, and is identified.

  The space information 700 (first dictionary data) created from the design information used in step S903 includes the robot arm 701, the work 710, and the tools 711 and 712 as shown in FIG. , Cable information is not included. However, for the cable, the connection list 900 (second dictionary data) described with reference to FIGS. 9A and 9B is prepared.

  In step S903, the separated object group includes the spatial information of the cable (620, 621). The spatial information of the cable (620, 621) corresponds to an object that does not exist in the spatial information 700 created in advance.

  As described above, the connection list matching processing in step S904 is executed on the assumption that an object that does not exist in the previously created spatial information 700 is extracted. This connection list matching processing is performed by the matching unit 514. In the connection list collation processing in step S904, an object that does not exist in the spatial information 700 created from the design information (such as a cable-like connected object 620 or 621) and a connection list 900 (second dictionary data: FIG. 9B) ) And are matched. Then, an object (connected object) in which the connection characteristic information (905, 906) corresponding to the connection position or the direction is matched is identified by this collation and identified.

  In the subsequent step S905 (virtual environment space information output processing: output processing step), the 3D model of the object that has been determined to be relevant in the matching in steps S903 and S904 (first matching processing and second matching processing, respectively) and is identified. Generate Then, it is registered in the virtual environment space information 530 (3D virtual environment).

  That is, for objects corresponding to the space information 700 (FIG. 7: first dictionary data) in step S903, a 3D model corresponding to those objects is generated and registered in the virtual environment space information 530. In this case, since the spatial information 700 (FIG. 7: first dictionary data) is CAD information generated from design information or the like, for example, CAD information corresponding to the object in the spatial information 700 is stored in the virtual environment space. By arranging the information in the information 530, the registration to the 3D virtual environment is completed.

  Also, as a result of the collation in step S904, in the spatial information input from the environment input device 501, for an object corresponding to the connection list 900 (second dictionary data: FIG. 9B), the spatial information of the object (3D model) is generated and registered in the virtual environment space information 530.

  Here, as one of the creation policies of the connection list 900 (second dictionary data: FIG. 9B), for example, there is the following. One of the creation policies of the connection list 900 is that the connection list 900 (second dictionary data: FIG. 9B) may be used, for example, when creating a robot operation program, and exists in the virtual environment space information 530. It is configured to include (only) the object to be performed. In this case, if there is an object corresponding to the connection list 900 (second dictionary data: FIG. 9B) as described above, the space information of all the objects is generated and the space information for the virtual environment is generated. 530.

  The different creation policies of the connection list 900 (second dictionary data: FIG. 9B) include both objects that should exist in the virtual environment space information 530 and objects that should not exist. There are things to configure. In this case, the object that should exist and the object that should not exist in the virtual environment space information 530 are registered in the connection list 900 together with a flag (not shown) for identifying both. When the connection list 900 is configured with such a policy, the space information is generated for only the objects that should exist in the virtual environment space information 530 by referring to the flag in the connection list 900, and the virtual environment space information 530 is generated. What is necessary is just to register in the space information 530.

  An object such as the cable (620, 621) needs to be handled as an object that should exist in the virtual environment space information 530, for example. For example, when creating a robot operation program using a virtual environment, a flexible object such as a cable (620, 621) has an undefined shape. Therefore, a space that may be occupied by a flexible object such as the cable (620, 621) needs to be explicitly displayed as a robot arm intrusion prohibition area when displaying the virtual environment space information 530 on the display. There is. A method of generating a space occupied by a flexible object such as the cables (620, 621) and registering the space in the virtual environment will be described later separately.

  As described above, of the objects existing in the spatial information input from the environment input device 501, as a result of matching with the spatial information 700 (first dictionary data) created in advance from the design information, It is registered in the environmental space information 530. In addition, as a result of collation with the connection list 900 (second dictionary data: FIG. 9B), among the objects existing in the space information input from the environment input device 501, the corresponding object is determined to be the virtual environment space information. 530 is registered.

  The control procedure in FIG. 10 includes first and second collation of 3D data input from the environment input device 501 with the spatial information 700 (first dictionary data) and the connection list 900 (second dictionary data). (S903, S904). Then, in the first and second matching processes (S903 and S904), the objects corresponding to the spatial information 700 (first dictionary data) and the connection list 900 (second dictionary data) are converted from the input 3D data into 3D data. Generate a model and register it in the 3D virtual environment. That is, in the virtual environment space information output processing (S905) in FIG. 10, the generation of this 3D model and the placement in the 3D virtual environment (virtual environment space information 530) are performed. This virtual environment space information output processing (S905) can be performed by the space information creation unit 515.

  Further, in step S906 in FIG. 10, the virtual environment space information 530 generated as described above is input to the robot operation program creation device 550. Then, the robot operation program creating device 550 can display the virtual environment whose environment has been recognized on the monitor by the virtual environment display unit 561 (FIG. 5) based on the virtual environment space information 530.

  Here, the user can operate the virtual environment display (equivalent to the space information 800 in FIG. 8) on the virtual environment display unit 561 using the user interface of the robot operation creation unit 562. At this time, the user interface of the robot operation creation unit 562 including various pointing devices, touch panels, and the like as described above can be used. This makes it possible to provide an intuitive user interface as if the object in the virtual environment display (equivalent to the spatial information 800 in FIG. 8) is directly operated.

  In this way, a user interface for creating the robot operation program 570 can be provided on the virtual environment display unit 561 using, for example, a 3D-displayed virtual environment. The user can easily and quickly program (teach) the operation of the robot apparatus 100 via an intuitive user interface for directly operating an object in the virtual environment (for example, bending the robot arm 801).

  In step S907 of FIG. 10, the robot operation program 570 created as described above is input to the robot controller 300, and the robot arm 200 is operated according to the robot operation program 570 (actual machine operation processing).

  Hereinafter, information processing performed by the control procedure of FIG. 10 will be described with reference to FIGS. Hereinafter, a detailed configuration or a modified example of the information processing in each step will also be described.

  FIGS. 11A, 11B, and 11C show processes of the environment recognition result input process (S901), the planar object extraction process (S902), and the object separation and matching process (S903) of FIG. Schematically shows spatial information to be obtained. Here, the processing target in FIGS. 11A to 11C is mainly the spatial information input from the environment input device 501.

  FIG. 11A shows spatial information (800) input from the environment input device 501 at the time of the environment recognition result input processing (S901 in FIG. 10). In FIG. 11A, for convenience, the spatial information (800: environment recognition result) in FIG. ) Are used with the same reference numerals (800). A plane 850 in the figure is an environment recognition result of the installation surface 650 in the real environment.

  FIG. 11B shows spatial information 800a obtained by adding a planar object extraction process (S902 in FIG. 10) to the spatial information 800 in FIG. 11A. It can be seen that 850 has been removed. Also, in the spatial information 800a of FIG. 11B, a plurality of objects are recognized as one lump, as in a group of recognized objects surrounded by a dotted line (860).

  FIG. 11C shows spatial information 800b obtained by adding the object separation and collation processing (903 in FIG. 10) to the spatial information 800a in FIG. 11B, and separates the spatial information 800b in units surrounded by dotted lines. (870-875). In FIG. 11 (c), the distance between the objects is large for the sake of simplification of the description, and the spatial information is not actually moved.

  FIGS. 12A and 12B schematically show details of the object separation and collation processing (S903 in FIG. 10). In FIG. 12A, objects (701, 710 to 712) existing in the space information 700 (first dictionary data) created in advance from the design information are indicated by broken lines. In the object separation and matching process (S903 in FIG. 10), these objects (701, 710 to 712) are separated by matching with the spatial information 700 (first dictionary data).

  However, in this example, the two cables indicated by solid lines and the recognition object group 860 including the work 710 do not correspond to the spatial information 700 (first dictionary data) and have not been recognized yet. For example, when the work 710 is placed at a position slightly different from the installation position on the space information 700 (first dictionary data), the space information 700 (first dictionary data) is simply used. It does not hit just by matching.

  Assuming such a case, when matching with the spatial information 700 (first dictionary data), matching (matching) processing including an allowable range as shown in FIG. Can be. FIG. 12B illustrates shapes of the work 710 and the recognition object group 860 in the spatial information 800 on the XY plane.

  In FIG. 12B, the position of the work 710 and the group of recognized objects 860 are shifted in the XY directions as indicated by arrows. Even in such a case, for example, the spatial information 700 (first dictionary data) is compared with the outline, and if the distance of the outline in the normal direction is equal to or less than a predetermined threshold, the spatial information 700 exists in the previously generated spatial information. It can be determined that the object to be performed. Of course, when there is no corresponding object in the spatial information 700 (first dictionary data), it is determined that there is no corresponding object. The threshold value of the matching (matching) process based on the contour comparison including the allowable range as described above can be specified in advance by the user by inputting a numerical value or the like.

  Although FIG. 12B shows the matching (collation) processing based on the contour comparison on the XY plane, the same matching (collation) processing can be performed for the ZX plane, the ZY plane, and the like. In this case, it goes without saying that the matching (collation) processing including the allowable range can be performed as described above.

  FIGS. 13A and 13B show a detailed example of the connection list collation processing (S904 in FIG. 10). The connection list matching process (S904 in FIG. 10) is an environment recognition process performed using the connection list 900 (second dictionary data: FIG. 9B) after the object separation and matching process (S903 in FIG. 10). As described above, the connection list 900 (second dictionary data: FIG. 9B) is included in the spatial information 700 (first dictionary data) like the connection object (cable: 820, 821) in FIG. It is created assuming that the unrecognized object is recognized in the environment.

  FIG. 13A shows an object group (820, 821) in the spatial information 800 in the object separation and collation processing (S903), similarly to FIG. 9A. The area to be recognized in the environment in the spatial information 800 is partitioned into a 6 × 4 grid.

  On the other hand, reference numerals 820a and 821a in FIG. 13B denote connection positions (905) and connection directions (906) in the connection list 900 (second dictionary data: FIG. 9B) illustrated in FIG. 9B. Is an arrangement area of a connection object (cable: 820, 821) corresponding to the definition. The arrangement regions 820a and 821a are regions in the grid where a connection object (cable: 820, 821) can exist according to the definition of the connection position and the connection direction (905, 906).

  Specifically, the arrangement regions 820a and 821a are arranged in a row of cells adjacent to the connection position (905) in the connection direction (906). For example, in the case of a connection (cable) connecting the objects A and B in FIG. 9B, the cells (3, 3) and the cells (3, 3) arranged on the left side of the cell (2, 3) and the right side of the cell (4, 2) Cell (3, 2) becomes arrangement area 820a.

  The arrangement areas 820a and 821a where such connected objects (cables: 820 and 821) can exist are created in advance by the CPU (301) by calculation according to the definition of the connection position and the connection direction (905 and 906). be able to. Alternatively, if the processing capacity of the CPU (301) is sufficient, the arrangement areas 820a and 821a may be generated in real time when performing collation with the connection list 900 (second dictionary data: FIG. 9B). .

  Then, when the object in the spatial information 800 is compared with the connection list 900 (second dictionary data) (S904), not only the connection position (905) and the connection direction (906) but also the arrangement areas 820a and 821a Determine the presence of an object and its continuity. This makes it possible to automatically and reliably determine whether or not the object (820, 821) is space information to be registered in the virtual environment space information 530.

  Although FIG. 13 shows only the arrangement areas 820a and 821a on the XY plane, for example, an arrangement area of an object such as a cable can be defined on a plane such as YZ or XZ. For example, as described with reference to FIG. 9, information in the height direction (Z) is recorded in the connection list information 900 in the connection list 900 (905). Then, using the information on the height direction (Z), an arrangement area of an object such as a cable in a space including the height direction (Z) is defined in the same manner as described above, and registered in the virtual environment space information 530. be able to.

  As described above, by using the connection list 900 (second dictionary data: FIG. 9B) of the present embodiment, the definition of the arrangement area of an object such as a cable can be defined as a three-dimensional (XYZ) space. . An example in which a three-dimensional (XYZ) space arrangement area is defined as, for example, an intrusion prohibition area will be described later with reference to FIG.

  In the example of FIG. 13, the object (820, 821) is determined to be a connected object (cable) included in the connection list 900 (second dictionary data: FIG. 9B), and the connected object (cable) is determined. It can be reliably registered in the virtual environment space information 530.

  As described above, according to the present embodiment, in the spatial information obtained from the installation environment 600 as shown in FIG. 6 by the environment input device 501, the spatial information 700 (FIG. 7: first dictionary data ) Can be registered in the virtual environment space information 530. In addition, an object corresponding to a connection list 900 (second dictionary data: FIG. 9B) prepared for a connected object such as a cable, for example, can also be registered in the virtual environment space information 530.

  Then, the virtual environment is displayed on the GUI using the virtual environment space information 530 by the robot operation program creating device 550, the robot arm or the work is operated via the GUI, and the robot teaching is performed through the operation. it can. Therefore, the user can quickly and reliably program (teach) the operation of the robot device 100 via a comfortable and intuitive user interface as if the user directly manipulates an object in the virtual environment.

  The virtual environment used at the time of teaching the robot is created based on spatial information actually obtained from the installation environment (600 in FIG. 6) using the environment input device 501. For this reason, it is possible to generate a GUI display of a virtual environment accurately reproduced in accordance with a robot apparatus to be actually operated and its installation environment, and perform robot programming using an accurate virtual environment with few errors. Can be.

  However, the virtual environment space information 530 does not need to be used only for robot programming through the above-described interactive operation of the user. For example, the robot operation program creation processing using the virtual environment space information 530 may be automatically performed based on some algorithm by the CPU without performing a user operation.

  As described above, according to the first embodiment, not only the collation processing with the spatial information 700 (FIG. 7: first dictionary data) created in advance from the design information, but also the connection list 900 (second dictionary data) 9 (b)) (connection list matching process: S904 in FIG. 10).

  That is, the first and second trials of trying to match the spatial information 1300 (3D data) obtained by the environment input device 501 with the spatial information 700 (first dictionary data) and the connection list 900 (second dictionary data), respectively. 2 collation steps (S903, S904). Then, in these first and second matching steps, a 3D model of an object corresponding to the first or second dictionary data in the 3D data is generated and arranged in the 3D virtual environment (output processing step: for virtual environment). Spatial information output processing (S905).

  For this reason, even if there is a connection object such as a cable that does not exist in the pre-created space information 700 (FIG. 7) but exists in the environment-recognized space information 800 (FIG. 8), the object will With such a simple process, the virtual environment space information 530 can be registered.

  Here, an example of the operation and effect of the first embodiment will be described with reference to FIG. FIGS. 14A to 14C show the installation surface 650 in the real environment where the robot arm 200 is installed.

  In the related art, the connection list 900 (second dictionary data: FIG. 9B) of the present embodiment is not used, and a virtual environment for robot programming is created by CAD processing or the like based on design information. Was.

  As described above, it is difficult to predict the actual arrangement shape and the like of a connection object such as a cable, and it is difficult to generate spatial information by CAD or the like. For this reason, conventionally, for an object such as a connection object (such as a cable), a user registers a large intrusion prohibited area such as a virtual obstacle 1000 on the virtual environment as shown in FIG. Had been done.

  Here, it is assumed that, when a large intrusion prohibition area such as the virtual obstacle 1000 in FIG. 14A is registered, the hand of the robot is moved from the teaching point p1 to p2. These teaching points are defined as the position and orientation of a reference position (part) set at the tip of the robot arm 200, for example.

  When performing the robot teaching by displaying the virtual environment corresponding to FIG. 14A on the GUI, there is a virtual obstacle 1000 that is relatively large between the teaching points p1 and p2. In order to avoid the virtual obstacle 1000, the user needs to place a taught point at a high position such as the taught point p3 between the taught points p1 and p2. By programming the teaching point p3 between the teaching points p1 and p2, a hand trajectory as shown by 1001 in FIG.

  On the other hand, in the first embodiment, spatial information close to the real environment as shown in FIG. 14B can be presented to the user. According to the configuration using the connection list 900 (second dictionary data: FIG. 9B) of the present embodiment, as described above, the definition of the arrangement area of an object such as a cable is defined as a three-dimensional (XYZ) space. Can be defined. The arrangement area of this three-dimensional space can be registered in the virtual environment space information 530 as an intrusion prohibition area.

  As described above, the connection list 900 (second dictionary data: FIG. 9B) is generated based on, for example, a cable pull-out position and a connector position. Therefore, by the spatial calculation in units of cells shown in FIG. 13 and the like, it is possible to register an extremely realistic three-dimensional space arrangement area having the smallest volume in the virtual environment space information 530 as an intrusion prohibition area.

  The virtual obstacle 1002 in FIG. 14B is generated using the connection list 900 (second dictionary data: FIG. 9B) as described above. The virtual obstacle 1002 in FIG. 14B can be generated as occupying a smaller space than the virtual obstacle 1000 in FIG. Therefore, the teaching point p3 that defines the avoidance operation of the robot arm 200 can be set at a position lower than that in FIG. 14A, and as a result, a hand trajectory like 1003 is obtained.

  As described above, according to the present embodiment, it is possible to create a shorter trajectory than in the related art in order to move the hand of the robot arm 200, so that the operation time of the robot can be reduced. If the robot apparatus according to the present embodiment is used in a manufacturing (production) environment for articles, the use of the virtual environment programming according to the present embodiment makes it possible to reduce the tact time and efficiently manufacture industrial products. It becomes possible to manufacture (produce) articles such as The present embodiment is considered to be highly effective particularly in a case where an object such as a cable is long and how to arrange the object is unknown, and in particular, in a case where the entry prohibition region of the robot arm 200 must be set large.

  For example, a comparison between FIG. 14A and FIG. 14B shows that the operation time of the robot can be shortened by about 8% as shown in FIG. Conceivable. In FIG. 15, the vertical axis represents the time required for a specific robot operation, 1501 corresponds to the operation time in the conventional (FIG. 14A) method, and 1502 corresponds to the operation time in the method of the present embodiment (FIG. 14B). .

  In FIG. 14B, the intrusion prohibition area is described as a rectangular parallelepiped, but the recognized connection object is fattened to generate an intrusion prohibition area such as the virtual obstacle 1004 in FIG. 14C. Can be considered. The virtual obstacle 1004 is generated by amplifying (thickening) the contour of an object (connecting object 620 such as a cable) in spatial information obtained from the installation environment (600 in FIG. 6) using the environment input device 501, for example. can do.

  In FIG. 14B of the first embodiment, the virtual obstacle 1002 has been described as being generated using the connection list 900 (second dictionary data: FIG. 9B).

  However, when the environment input device 501 is configured by a 3D camera or the like, a 3D model can be generated using the spatial information actually obtained from the installation environment (600 in FIG. 6). For example, the generation of the 3D model can be performed using the spatial information of the object separated by the object separation and collation processing (S903).

  Thus, the current 3D model of an object such as a connected object (for example, a cable: 620 or 621 in FIG. 6) is created by using the space information actually obtained from the installation environment (600 in FIG. 6) by the environment input device 501. Can be generated. Here, the connection object (for example, a cable) is registered only in the connection list 900 (second dictionary data), and the shape information is not included in the connection list 900 (second dictionary data).

  On the other hand, with respect to the robot arm, the work, and the jigs and tools (710 to 712 in FIG. 7), the space information 700 (FIG. 7: first dictionary data) created from the design information (CAD data and the like) is used for the object. A current 3D model can be generated.

  In this case, a particular problem is that the cable that is deformed as the position of the workpiece or the jig (710 to 712 in FIG. 7) is arranged at a position slightly different from the work protocol or the design specification. (620 and 621 in FIG. 6).

  In the second embodiment, an example of processing in a case where an arrangement space of an object such as a deformable cable is treated as a virtual obstacle will be described.

  FIGS. 16A and 16B show a work 1100, a work 1101, and a connection (for example, a cable) 1102 which are respectively arranged at different positions on the installation surface 650 of the robot arm in the real environment. ing. In FIGS. 16A and 16B, the work 1100 is arranged at different positions, and accordingly, the connection object (for example, a cable) 1102 is deformed into a different shape. Along with this, the shape of the arrangement space of, for example, a rectangular parallelepiped connection object 1102 to be handled as virtual obstacles 1103 and 1104 is different.

  As an example in which two different cases occur, the following situation can be considered. For example, one of the virtual obstacles 1103 and 1104 in FIGS. 16A and 16B, for example, the virtual obstacle 1103 (FIG. 16A) is connected to the connection list 900 (second dictionary data: FIG. 9B It is assumed that the image is generated in the shape shown in FIG.

  However, when the 3D data of the connected object (for example, a cable) 1102 is generated using the spatial information actually obtained by the environment input device 501, the 3D data has a shape as shown in FIG. An object 1104 is generated in a shape as shown.

  Also, for some reason, the 3D data of the connected object (for example, a cable) 1102 is generated twice (or more) using the spatial information actually obtained by the environment input device 501, during which the work 1100 is moved. In such a case, the above inconsistency arises.

  These virtual obstacles 1103 and 1104 are the space occupied by the connection object (for example, the cable) 1102, and both can be generated as a rectangular parallelepiped shape having, for example, the size in the XYZ directions of the cable arrangement space. The generation of the 3D model of the space occupied by the connection object (cable) 1102, that is, the space corresponding to the virtual obstacles (1103 to 1104) in the robot control is performed by outputting the virtual environment space information in FIG. Can be run with

  If the occupied space (virtual obstacles 1103 to 1104) before and after the deformation of the connected object (cable) 1102 is generated in the virtual environment space information output processing (S905) as described above. The CPU (301) further performs the following processing.

  FIG. 16C illustrates an example of a process of generating a new virtual obstacle 1105 by integrating different virtual obstacles 1103 and 1104 of the connection object (for example, a cable) 1102 generated as described above. ing. Here, the CPU (301) generates a 3D model of the occupied space obtained by integrating the space (1103, 1104) occupied by the connected object 1102 before and after the deformation of the connected object 1102. Then, it is arranged in the 3D virtual environment (virtual environment space information 530) as a space virtual obstacle 1105 obtained by this integration.

  In the processing related to the three-dimensional shape in FIG. 16C, the shape of the virtual obstacle 1105 is determined using the maximum value of the (maximum) size of the rectangular parallelepiped virtual obstacles 1103 and 1104 in the X, Y, and Z axis directions. (Cuboid). Similar integration processing by shape calculation can be applied by the CPU (301) even when there are three or more target virtual obstacles.

  As described above, when a plurality of virtual obstacles 1103 and 1104 are generated from the arrangement space of the connection object (for example, the cable) 1102, the virtual obstacles 1103 and 1104 are integrated using, for example, the above-described calculation of the maximum value of the three-dimensional size. Then, a virtual obstacle 1105 can be generated. The generated virtual obstacle 1105 can be registered in the virtual environment space information 530 as an intrusion prohibited area. For this reason, it is possible to cope with a case where the connected object is deformed (possibly) due to a time difference or due to a difference in situation from when the connection list 900 (second dictionary data: FIG. 9B) was created. . This makes it possible to appropriately set the robot arm intrusion prohibition area. That is, according to the present embodiment, by integrating the spatial information actually obtained by the environment input device 501, more reliable virtual environment space information 530 can be generated and used for robot programming. According to the second embodiment, the virtual environment space information 530 with high validity can be generated, and highly reliable robot control can be performed without interference of the robot arm.

  FIG. 6 of the first embodiment shows an example of a robot arm installation environment 600 in a real environment. The arrangement matches spatial information 700 (FIG. 7: first dictionary data) generated based on the design information as shown in FIG. 7 (except for the connected objects 620 and 621). In FIG. 6, in addition to the robot arm 200, the work 610 and the jigs 611 to 612 (FIG. 6) are arranged at corresponding positions if the space information 700 (FIG. 7: first dictionary data) is almost collated. I have.

  However, in the actual installation environment (1200), as shown in FIG. 17, an object existing in the spatial information 700 (FIG. 7: first dictionary data) in FIG. There is. Such misplacement and work mistakes tend to occur due to forgetting to install when there are many work pieces.

  FIG. 17 shows the spatial information 1200 obtained by recognizing the environment from the spatial information actually obtained by the environment input device 501, similarly to FIG. The difference from FIG. 8 is that there is no object corresponding to the work 812 (FIG. 8). The work object (812) is not arranged due to misplacement, misoperation, or the like, and is missing in the spatial information 1200. In FIG. 17, the object not measured by the environment recognition is shown as a work 1212 (broken line).

  The space information 1200 whose environment is recognized as shown in FIG. 17 is generated based on the space information 700 (FIG. 7: first dictionary data) in FIG. In FIG. 17, the cable processed by the connection list 900 (second dictionary data) is not shown.

  In the third embodiment, as shown in FIG. 17, when environment recognition is performed using the spatial information 700 (FIG. 7: first dictionary data) in FIG. Related to information processing when an object is missing.

  The state in FIG. 17 is a state in which the work 712 existing in the space information 700 in FIG. 7 is not arranged in the real environment and is missing. Assuming such a state, a control procedure as shown in FIG. 18 can be considered.

  FIG. 18 corresponds to a modification of the control procedure of FIG. 10 of the first embodiment. The difference from FIG. 8 is that the installation omission determination (S908) and the installation omission warning process (S909) are the object separation and collation processing (S903). And the connection list collation processing (S904). Steps other than the installation omission determination (S908) and the installation omission warning process (S909) are the same as those in FIG. 10, and a detailed description thereof will be omitted below.

  In steps S901 to S903 in FIG. 18, environment recognition is performed on the spatial information actually obtained by the environment input device 501 as in FIG. 10. In particular, in the object separation and collation processing (S903), the object separation of the spatial information actually obtained by the environment input device 501 is performed, and the space information 700 (first dictionary data) is used to set the robot arm installation surface 650. The spatial information of the currently installed object is generated.

  In the installation omission determination (S908) of FIG. 18, the result of the object separation and collation processing (S903) is evaluated. Here, it is determined whether or not all the objects included in the spatial information 700 (first dictionary data) have been identified by the object separation and collation processing (S903). Specifically, spatial information 1200 (FIG. 17), which is a result of separation and collation for each object in the object separation and collation processing (S903) and environment recognition (identification), is compared with spatial information 700 created in advance. Then, it is determined whether or not there is an object that exists in the space information 700 created in advance and does not exist in the space information 1200 recognized as the environment.

  In the state of FIG. 7, since the work 712 present in the space information 700 is not arranged in the real environment, the installation omission warning process (S909) is executed. If the work 712 existing in the space information 700 is arranged without omission in the real environment, the process directly proceeds from step S908 to S904.

  In the installation omission warning process (S909), there is an object that exists in the space information 700 (FIG. 7: first dictionary data) and does not exist in the environment-recognized space information 1200 by character information display or audio display. The user is notified of warning information indicating that there is a leak. At this time, for example, a display output device such as a monitor (display) provided in the environment recognition device 500 or an audio output device can be used. At this time, a warning message by display or voice may be "Work xxx is not installed. Please check". At this time, an image near the object or a 3D image of the object generated from spatial information may be displayed simultaneously with the warning message. By executing such a dialog, the user can perform an operation such as removing a misplaced work or a tool.

  As described above, according to the third embodiment, when there is an object that exists in the spatial information 700 (FIG. 7: first dictionary data) created in advance and does not exist in the environment-recognized spatial information 1200, By giving a warning to, installation omission can be avoided. That is, when an object included in the spatial information 700 (FIG. 7: first dictionary data) created in advance does not exist in the 3D data acquired by the 3D data acquisition device, a warning to warn the user to that effect. Notify information. As a result, registration omission in the virtual environment space information 530 can be avoided, and obstacles such as creation of unpractical robot teaching data without an object that should exist during programming can be avoided.

  Fourth Embodiment In a fourth embodiment, processing that is performed in a case where an object that does not exist in the created spatial information 700 (FIG. 7: first dictionary data) exists in the environment-recognized spatial information, contrary to the third embodiment, will be described.

  FIG. 19 shows spatial information 1300 obtained by recognizing the environment from the spatial information actually obtained by the environment input device 501, similarly to FIG. 8 in that an object 1313 (such as a hammer) is present.

  Here, it is assumed that the object 1313 is not included in the spatial information 700 (FIG. 7: first dictionary data) in FIG. On the other hand, the spatial information 1300 in FIG. 19 is compared with the spatial information 700 in FIG. 7 (FIG. 7: first dictionary data), and the 3D of the robot arm 1301 and the work 1310 to 1312 arranged on the installation surface 1350 is obtained. Data is being generated. At the same time, the spatial information 1300 in FIG. 19 includes the spatial information of an unmatched (not identified by matching) object 1313 among the objects separated in the object separation processing (S903 in FIG. 10).

  The object 1313 (such as a hammer) in FIG. 19 is not included in the connection list 900 (second dictionary data), unlike the connection object (such as a cable) in the first embodiment. Such a case may occur when there is a misplaced tool in the assembly environment, or when a new work piece that is not assumed in the prior design is added.

  In the fourth embodiment, as shown in FIG. 19, an object that is not included in any of the spatial information 700 (FIG. 7: first dictionary data) and the connection list 900 (FIG. 9: second dictionary data) It relates to information processing when it exists. Assuming such a state, a control procedure as shown in FIG. 20 can be considered.

  FIG. 20 corresponds to a modified example of the control procedure of FIG. 10 of the first embodiment. The difference from FIG. 8 is that the obstacle determination (S910) and the obstacle warning process (S911) are virtually the same as the connection list matching process (S904). This is a point inserted during the environment space information output processing (S905). Steps other than the connection list collation processing (S904) and the virtual environment space information output processing (S905) are the same as those in FIG. 10, and a detailed description thereof will be omitted below.

  As described with reference to FIG. 10, in steps S901 to S904 in FIG. 20, the CPU (301) converts the spatial information actually obtained by the environment input device 501 into spatial information 700 (FIG. 7: first dictionary data), and It is checked against the connection list 900 (FIG. 9: second dictionary data).

  In the obstacle determination (S910), the spatial information (1300) obtained by the environment input device 501 includes the spatial information 700 (FIG. 7: first dictionary data) and the connection list 900 (FIG. 9: second dictionary data). ) Is determined whether there is an object (obstacle) not included in any of the above. If such an object exists in the space information 1300 obtained by the environment input device 501, it is determined that it is an obstacle, and the process proceeds to the obstacle warning process (S911). The process proceeds to the spatial information output process (S905).

  In the obstacle warning process (S911), an object that does not exist in neither the space information 700 nor the connection list 900 exists in the environment-recognized space information 1300 by character information display or audio display, that is, that there is an obstacle. Notify the user of warning information. At this time, for example, a display output device such as a monitor (display) provided in the environment recognition device 500 or an audio output device can be used. At this time, the warning message by display or voice may be "There is an unconfirmed object (object that cannot be recognized in the environment. Please confirm it)". Further, a warning message including a work instruction such as "obstruction is present" or "obstruction exists. Please remove" may be generated more explicitly. In response to this, the user can perform an operation such as removing the displayed obstacle if the obstacle is unnecessary for the assembly environment.

  Note that the processing of the present embodiment can be compatible with the processing of forgetting to install in the third embodiment. That is, the flows of FIG. 18 and FIG. 20 can be combined, and both steps S908, S909 and S910, S911 can be implemented in the same control procedure.

  As described above, according to the fourth embodiment, when an object 1313 that is neither present in the spatial information 700 nor in the connection list 900 exists in the spatial information 1300 obtained by the environment input device 501, that is, when there is an obstacle. Can generate warning information to that effect.

  That is, according to the fourth embodiment, the spatial information 1300 (3D data) obtained by the environment input device 501, the spatial information 700 (first dictionary data), and the connection list 900 (second dictionary data) It includes first and second matching steps for respectively trying matching. If the spatial information 1300 (3D data) obtained by the environment input device 501 includes an unverified object that does not correspond to any of the first and second dictionary data in these steps, Is notified of the warning information.

  This allows the user to remove an object that is (probably) an obstacle from the site. Even if the robot teaching is performed using only the spatial information registered in the spatial information for virtual environment 530, the robot arm and the object can be removed. Interference can be avoided.

  In the fourth embodiment, when an object that is neither present in the space information 700 created from the design information nor in the connection list 900 exists in the spatial information 1300 obtained by the environment input device 501, that is, when there is an obstacle, Has shown the example which generates warning information.

  The control according to the fourth embodiment is based on the recognition that an object that does not exist in neither the spatial information 700 nor the connection list 900 is an obstacle. However, there is a case where a new work that is not assumed in the prior design is added due to a change in the work protocol or a change in the specification, etc. On the other hand, in response to such a change, the spatial information 700 (or the connection list) is added. 900) may not have been updated.

  Assuming such a situation, it may not be sufficient to determine that an object that does not exist in neither the spatial information 700 nor the connection list 900 is an obstacle that should be removed immediately. .

  Therefore, assuming a case in which the addition of an object such as a work object due to a change in a work protocol or a change in specifications and the update of the spatial information 700 (or the connection list 900) are not synchronized, in such a case, It may be desirable to contact the user. In this case, for example, control is performed to inquire the user whether or not to register at least the object in the virtual environment space information 530.

  FIG. 21 shows space information 1400 whose environment has been recognized in the same manner as FIG. The difference from FIG. 19 is that an object 1414 exists instead of the object 1313. The object 1414 according to the present embodiment is not the object 1313 (such as a misplaced hammer), but is a work (a work or the like) added due to, for example, a change in work rules or a change in specifications.

  Here, it is assumed that the spatial information 700 (FIG. 7: first dictionary data) in FIG. 7 does not keep up with the update due to the addition of the object 1414, and the spatial information 700 does not include the object 1414. . On the other hand, the spatial information 1400 in FIG. 21 is compared with the spatial information 700 in FIG. 7 (FIG. 7: first dictionary data), and the 3D of the robot arm 1401 and the work objects 1410 to 1412 arranged on the installation surface 1450 is obtained. Data is being generated. At the same time, the spatial information 1400 in FIG. 21 includes the spatial information of the uncollated object 1414 among the objects separated in the object separation processing (S903 in FIG. 10).

  The object 1414 in FIG. 21 is not included in the connection list 900 (second dictionary data), unlike the connection object (such as a cable) in the first embodiment.

  However, the object 1414 is not merely a misplaced tool or the like, but may be a work required in the work process performed by the robot arm 1401 as described above. Therefore, assuming such a case, it is desirable to inquire the user when the spatial information of the object 1414 is recognized.

  In order to inquire the user about such an object 1414, for example, the control procedure of FIG. 20 of the fourth embodiment may be modified as follows (the steps other than those described below are the same as described above). For these, see Example 4 above).

  Since the object 1414 as described above is an object that does not exist in the spatial information 700 or the connection list 900, the determination in step S910 in FIG. 20 results in “obstruction present”, and a branch to step S911 occurs. .

  In the case of the fifth embodiment, in the obstacle warning process of step S 911, the registration to the virtual environment space information 530 is performed, for example, instead of simply warning the existence of the obstacle (or suggesting the removal thereof). Ask if you want to do it. In this case, for example, by using a display output device such as a monitor (display) of the environment recognition device 500 or a sound output device, a dialog for inquiring whether to register in the virtual environment space information 530 is executed, and a user response is performed. To get. In such a case, a dialog message such as "There is an unconfirmed object (an object whose environment cannot be recognized). Would you like to register it in a virtual environment for teaching robots?"

  Alternatively, a dialog message such as “Operate the OK button when registering in the virtual environment for teaching robots, or the NO button when removing an object without registering” may be used. In this case, the user performs a response operation using a pointing device (or a keyboard) such as a mouse.

  In step S911, by performing the above-described user interface, the user can determine whether or not to register the object 1414 in the virtual environment space information 530 by a dialog response. Other controls and modified examples are the same as those in the fourth embodiment, and a description thereof will not be repeated.

  As described above, according to the fifth embodiment, when an object 1414 that does not exist in neither the spatial information 700 nor the connection list 900 exists in the spatial information 1400 (3D data) obtained by the environment input device 501, an inquiry is made. It can be carried out. In this inquiry, for example, an inquiry is made as to whether to register the virtual environment space information 530.

  That is, according to the fifth embodiment, the method includes the first and second 3D model generation steps of trying to match the spatial information 700 (first dictionary data) with the connection list 900 (second dictionary data). . In these steps, when the spatial information 1400 (3D data) obtained by the environment input device 501 includes an unmatched object that does not correspond to any of the first and second dictionary data, Perform the user interface step of inquiring. In this user interface step, the user is asked whether or not to generate a 3D model of the uncollated object and place it in the 3D virtual environment (space information for virtual environment 530).

  Accordingly, if the uncollated object 1414 is necessary for robot work, for example, if it is a work object, the user adds the space information of the object 1414 to the space information 530 for the virtual environment and uses it to perform necessary robot teaching. I can do it. If the object 1414 is, for example, a misplaced tool or the like, it is possible to prompt the user to remove the object 1414 from, for example, the installation surface 1450 as in the fourth embodiment.

  In the above description, the dialog performed in step S911 is described as inquiring whether to register the virtual environment space information 530. In addition to this, it is conceivable to perform control to extract necessary information from the spatial information of the object 1414 obtained by the environment input device 501 and additionally register the extracted information in the spatial information 700 (or the connection list 900). A user interface for inquiring whether or not to perform such additional registration can be implemented in the same manner as described above.

  In the present embodiment, a different configuration (900a in FIG. 22B) of the connection list 900 described in FIG. 9B of the first embodiment is shown.

  FIGS. 22 (a) and 22 (b) are illustrations of a mode equivalent to FIGS. 9 (a) and 9 (b) of the first embodiment. FIG. 22 (a) is equivalent to FIG. 9 (a), and shows the space information 700 obtained from the design information in FIG. Objects A to D (901 to 904) in FIG. 22A are equivalent to those in FIG. 9A, and objects A, B (901, 902) and C (903) correspond to the work objects 710 to 712 in FIG. , And D (904) corresponds to the robot arm 701. Further, as shown by the broken lines, the connection mode of the cable connecting between the objects A and B (901 and 902) and between the objects C and D (903 and 904) is also the same as that in FIG.

  In the robot installation environment as described above, FIG. 22A illustrates a state in which an obstacle (like a hammer) 1500 is further arranged. The obstacle 1500 in FIG. 22A is arranged so as to overlap on a cable connecting the objects C and D (903 and 904) as shown.

  As described above, the connection list 900 (FIG. 9B) described in the first embodiment mainly includes the connection position information (905) and the information on the connection direction (906). ) Does not include information on the shape or size.

  For this reason, as in the case of the obstacle 1500 in FIG. 22A, if the direction in which the object extends and the posture are similar to the connection position information (905) and the information about the connection direction (906), the possibility of erroneous recognition is low. can not deny.

  For example, consider a case where the obstacle 1500 overlaps a connection object (broken line: cable) in the posture as shown in FIG. In this case, when collation is performed with the connection position information (905) and the connection direction information (906) in FIG. 9B, there is a possibility that the object C and D on the second line in FIG. There is.

  In consideration of such a case, a configuration of the connection list 900a as shown in FIG. The connection list 900a in FIG. 22B includes information 907 on the volume of the object and information 908 on the aspect ratio of the object in addition to the connection position information (905) and the information on the connection direction (906) in FIG. Has been added.

  The connection list 900a in FIG. 22B defines the connection object (cable) in FIG. 22A, and the values of the connection position information (905) and the information on the connection direction (906) are shown in FIG. 9B. Is equivalent to

  Information 907 relating to the volume of the object and information 908 relating to the aspect ratio of the object are determined according to the size and size of the connection object (cable) arranged in the installation environment. Information 907 relating to the volume of the object is defined in the field on the left in FIG. The information 908 relating to the aspect ratio of the object is defined in the three fields on the left side of the drawing, for example, by the size ratio (X / Y, X / Z, Y / Z: no unit) in each coordinate axis direction. . In the fields on the right side of these pieces of information 907 and 908, tolerances regarding the volume value and the size ratio are defined.

  The value of each field of the information 907 related to the volume of the object and the information 908 related to the aspect ratio of the object in FIG. 22B is a representation of the object such as a connected object (cable). The values of these pieces of information 907 and 908 allow for identification of objects such as connections (cables), while not matching (matching) obstacles 1500 (like a hammer), for example.

  Collation with the connection list 900a in FIG. 22B can be performed in the connection list collation processing (S904) in FIG. 10 (Example 1) or FIG. 20 (Examples 4 and 5). In this case, the spatial information (3D data) of the uncollated object (1500) among those separated in the object separation processing (S903 in FIG. 10) is used for the collation. The collation with the information of the connection position (905) and the connection direction (906) is executed by the CPU (301) in the same manner as described in the first embodiment.

  Similarly, from the spatial information (3D data) of the object (1500), the CPU (301) calculates the volume of the object and the aspect ratio in each axis direction. Then, the calculated volume of the object (1500) and the aspect ratio in each axial direction are compared with information 907 on the volume of the object and information 908 on the aspect ratio of the object. In this comparison, differences within the allowable range defined in the information 907 and 908 are allowed.

  If all of the information (905, 906, 907, and 908) matches, the connection list matching process (S904) determines that the search is successful (hit: applicable). The other steps in FIG. 10 (Embodiment 1) or FIG. 20 (Embodiments 4 and 5) can be executed in the same manner as described above.

  As described above, the connection list 900a is configured as shown in FIG. 22B, and the above-described collation processing (S904 in FIGS. 10 and 20) is performed. As a result, it is possible to prevent the obstacle 1500, which is arranged at the center of the connection object (cable) as shown in FIG. Therefore, according to the connection list 900a, only the connection object such as an intended cable can be registered in the virtual environment space information 530 as being matched.

  If it is determined that there is no match in the comparison with the connection list 900a (S904), the configuration of the fourth or fifth embodiment can reliably execute the above-described warning or inquiring process regarding the obstacle. .

  As described above, according to the sixth embodiment, for the connection list 900a (second dictionary data), regarding the object such as the connection object that may be arranged in the installation environment, the shape or the size of the object. Includes information. In particular, for an object such as a connection object that may be placed in the installation environment, the volume or aspect ratio of the object or tolerance information relating to the volume or aspect ratio is included. For this reason, it is possible to prevent the obstacle (1500), which is arranged at the center of the connection object (cable), from being erroneously recognized. Therefore, according to the connection list 900a (second dictionary data), it is possible to register only the connection object such as the intended cable in the virtual environment space information 530 as matching.

  Therefore, according to the sixth embodiment, it is possible to obtain the virtual environment space information 530 that reproduces the actual robot installation environment with high reliability, and performs accurate and reliable robot teaching (programming) using this. be able to.

  The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  100: Robot device, 200: Robot arm (articulated robot), 201 to 206: Actuator, 211 to 216: Electric motor, 221 to 226: Reduction gear, 230: Servo control device (drive control unit), 300: Robot control Device, 330: program, 500: environment recognition device, 501: environment input device, 520: space information, 522: recording disk (recording medium), 530: space information for virtual environment, 532: recording disk (recording medium), 550 ... Robot operation program creation device 570 ... Robot operation program 580 ... Recording disk (recording medium).

Claims (17)

The control device generates a 3D model of the object installed in the installation environment using the 3D data acquired by the 3D data acquisition device from the installation environment of the production device, and arranges the 3D model in a 3D virtual environment corresponding to the installation environment of the production device. Virtual environment creation method
A first matching step in which the control device matches 3D data acquired by the 3D data acquisition device with first dictionary data including shape information of an object placed in the installation environment;
A second collation step in which the control device collates the 3D data acquired by the 3D data acquisition device with second dictionary data including information on a flexible object connected to an object placed in the installation environment. When,
Wherein the controller, based on the first or second verification process, look including an output processing step, which generates a 3D model of the object being disposed in the installation environment to place the 3D virtual environment,
The second dictionary data includes connection characteristic information including at least one of a position and a direction in which the flexible object is connected to an object placed in the installation environment, and the control device includes: A virtual environment creation method , wherein in the collation step, the 3D data acquired by the 3D data acquisition device is collated with the connection characteristic information .   2. The virtual environment creation method according to claim 1, wherein the first dictionary data includes data created based on design information of the production device. 3. The virtual environment creating method according to claim 1 , wherein the control device, when generating a 3D model corresponding to the flexible object in the installation environment in the output processing step, before and after the deformation of the 3D model. Generating a 3D model of the occupied space obtained by integrating the occupied spaces, and arranging the 3D model in the 3D virtual environment. 3. The virtual environment creation method according to claim 1 , wherein the second dictionary data includes an object that is likely to be arranged in the installation environment or the flexible object that connects the objects to each other. 3. A method for creating a virtual environment, characterized by including a volume of a flexible object, an aspect ratio, or permissible error information relating thereto. In the virtual environment creation method according to claim 1 in any one of 4, if the object included in the first dictionary data is not present in the 3D data obtained by the 3D data acquisition unit, A virtual environment creation method, wherein the control device notifies a user of warning information for warning the user. The virtual environment creation method according to any one of claims 1 to 5 , wherein the first and second matching steps are performed on the 3D data acquired by the 3D data acquisition device in the first and second matching steps. When a non-collated object that does not correspond to any of the second dictionary data is included, the control device notifies the user of warning information to warn the user of the unmatched object. 7. The virtual environment creating method according to claim 6 , wherein, after the notification of the warning information, the control device generates a 3D model of the uncollated object and informs the user whether or not to generate the 3D model and place the 3D model in the 3D virtual environment. A method for creating a virtual environment, comprising a user interface step of inquiring. And claim advance virtual environment creation method according to 1 to 7 any one of the production device is a robotic device of the articulated, with 3D data obtained by the 3D data acquisition device from the installation environment of the robot device Wherein the control device generates a 3D model of the object installed in the installation environment and arranges the 3D model in a 3D virtual environment corresponding to the installation environment of the robot device. The virtual environment creation method according to any one of claims 1 to 8 , wherein the flexible object is a cable. In the virtual environment creation method according to any one of claims 1 to 9, in the output processing step, the virtual environment creation method and outputting the 3D virtual environment on a display device. Control method for a robot apparatus, characterized in that to create a robot control program of the robot apparatus using the 3D virtual environment created by the virtual environment creation method according to claims 1 to 10. The method of controlling a robot device according to claim 11 , wherein the robot device is operated in an installation environment of the robot device according to the robot control program. A method of manufacturing an article, comprising: operating the robot apparatus in an installation environment of the robot apparatus according to the robot control program according to claim 11 , and manufacturing an article by the robot apparatus. A robot system comprising the robot device according to any one of claims 11 to 13 and the control device. A virtual environment creation program for causing the control device to execute the respective steps of the method for controlling a robot device according to claim 1 . A computer-readable recording medium storing the virtual environment creation program according to claim 15 . An information processing apparatus for generating a 3D virtual environment corresponding to an installation environment of a production device and displaying the 3D virtual environment on a display device,
Acquiring means for acquiring 3D data from a setting environment of the production device by a 3D data acquiring device;
First matching means for matching the acquired 3D data with first dictionary data including shape information of an object placed in the installation environment;
A second matching unit that matches the 3D data acquired by the 3D data acquisition device with second dictionary data including information on a flexible object connected to an object placed in the installation environment;
Based on the comparison result by said first or second comparing means, seen including output means, the outputting of the 3D virtual environment showing the arrangement of the object being disposed in the installation environment on said display device,
The second dictionary data includes connection characteristic information including at least one of a position and a direction in which the flexible object is connected to an object arranged in the installation environment, and the second collation unit includes: An information processing apparatus for collating 3D data acquired by a 3D data acquisition apparatus with the connection characteristic information .

Images