WO2024154218A1

WO2024154218A1 - Programming device

Info

Publication number: WO2024154218A1
Application number: PCT/JP2023/001146
Authority: WO
Inventors: 寛之米山
Original assignee: ファナック株式会社
Priority date: 2023-01-17
Filing date: 2023-01-17
Publication date: 2024-07-25

Abstract

The present invention generates an industrial machine program for generating a machining line which depicts an arbitrary and complex trajectory on a surface of a workpiece, and performing machining work along the trajectory on the basis of the generated machining line.　This programming device generates a program by which respective models for an industrial machine, a tool, a visual sensor, and a workpiece are disposed in a virtual space, and the industrial machine model performs work on the workpiece model by means of the tool model, wherein: a label on which an outline is depicted is disposed on the surface of the workpiece model; a captured image of the label at a position at which the visual model is disposed is generated on the basis of image-capturing parameters set to the visual sensor model; an outline is detected by executing image processing on the captured image; a machining line is generated on the basis of the detected outline; a normal vector to the surface of the workpiece model on the generated machining line is calculated; and the program is generated on the basis of the machining line and the normal vector.

Description

Programming Device

This disclosure relates to a programming device that generates programs for industrial machinery.

There exists a technology in which three-dimensional models of a robot equipped with a tool, a workpiece, and at least one peripheral device are arranged on a screen and displayed simultaneously, a processing line is specified on the three-dimensional model of the workpiece, a motion type, speed, position, and posture of a teaching point generated based on the specified processing line is specified, and an operation program for the robot to perform processing of the workpiece based on the specified processing line and the specified motion type, speed, position, and posture.
In addition, in a robot system having a robot, visual sensor, and workpiece within a working space, a technology exists in which a robot model of the robot, a visual sensor model of the visual sensor, and a workpiece model of the workpiece are placed in a virtual space that represents the working space in three dimensions, the workpiece model is measured using the visual sensor model, and a simulation is performed in which the robot model performs work on the workpiece model.
For example, as a method for specifying a processing line on a three-dimensional model of a workpiece, a method is known in which a processing line is specified by detecting a shape feature consisting of a contour line or a surface of a basic shape including a circle and a polygon on the three-dimensional model of the workpiece, or a shape formed by combining a plurality of basic shapes, or a portion where three-dimensional models of the workpiece contact each other during welding, etc. For example, see Patent Documents 1 and 2.
Alternatively, a method is known in which a three-dimensional shape including a curved surface or a three-dimensional shape including a plurality of continuous planes is filled with a motion pattern consisting of a continuous trajectory indicating a periodic motion of a tool, the three-dimensional shape is arranged in a virtual space so that the motion pattern is projected onto at least one surface of a work model, and the motion pattern is projected onto at least one surface of the work model to create a machining path for the tool.

JP 2017-140684 A JP 2019-48358 A JP 2013-248677 A

However, all of the robot's motion programs must be taught manually in order to perform machining work on complex trajectories. This makes the teaching process very time-consuming and labor-intensive, and requires the skilled operator.

Therefore, it is desirable to generate a machining line that traces an arbitrary and complex trajectory on the surface of a workpiece, and to generate a program for an industrial machine to perform machining work along that trajectory based on the generated machining line.

One aspect of the programming device disclosed herein is a programming device that places an industrial machine, a tool attached to the industrial machine, a visual sensor, and a model of a workpiece in a virtual space, and generates a program in which the industrial machine model performs an operation on the workpiece model using the tool model, places a label with a contour line on the surface of the workpiece model, generates a captured image of the label at the position where the visual sensor model is placed based on imaging parameters set for the visual sensor model, detects the contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the workpiece model, and generates the program based on the processing line and the normal vector.

One aspect of the programming device disclosed herein is a programming device that generates a program for an industrial machine, a tool attached to the industrial machine, a visual sensor, and a workpiece arranged in real space, in which the industrial machine performs an operation on the workpiece using the tool, and the programming device places a label with a contour line on the surface of the workpiece, acquires an image of the label captured at the position where the visual sensor is arranged based on imaging parameters set for the visual sensor, detects the contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the workpiece, and generates the program based on the processing line and the normal vector.

1 is a functional block diagram showing an example of a functional configuration of a programming device according to a first embodiment. FIG. 2 is a diagram illustrating an example of an industrial machinery system model in a virtual space. FIG. 13 is a diagram showing an example in which a label is placed on a surface of a work model. FIG. 13 is a diagram illustrating an example of a positional relationship between a visual sensor model and a label. FIG. 13 is a diagram showing an example of three-dimensional points detected on a contour line. FIG. 13 is a diagram showing an example of a generated processing line. 13 is a diagram showing an example of a normal vector relative to a surface of a workpiece model calculated at each point on a generated processing line; FIG. FIG. 11 is a diagram showing an example of set teaching points. FIG. 11 is a diagram illustrating an example of a simulation of a generated robot program. 11 is a flowchart illustrating a program generation process of the programming device. FIG. 11 is a functional block diagram showing an example of the functional configuration of a programming device according to a second embodiment. FIG. 13 is a diagram showing an example of labels for contours that intersect with each other at at least one point. FIG. 13 is a diagram illustrating an example of a positional relationship between a visual sensor model and a label. FIG. 13 is a diagram showing an example of a plurality of line segments obtained by dividing a contour line; FIG. 13 is a diagram showing an example of a result of combining a plurality of divided line segments. FIG. 13 is a diagram showing an example of three-dimensional points detected on a contour line. FIG. 13 is a diagram showing an example of a generated processing line. 13 is a diagram showing an example of a normal vector relative to a surface of a workpiece model calculated at each point on a generated processing line; FIG. FIG. 11 is a diagram showing an example of set teaching points. 11 is a flowchart illustrating a program generation process of the programming device. FIG. 13 is a functional block diagram showing an example of the functional configuration of a programming device according to a third embodiment. 11 is a flowchart illustrating a program generation process of the programming device. FIG. 13 is a diagram showing an example of a visual sensor that is fixedly disposed separately from a robot.

First Embodiment
First, an outline of this embodiment will be described. In this embodiment, in a virtual space in which an industrial machine, a tool attached to the industrial machine, a visual sensor, and a model of a workpiece are respectively arranged, a programming device arranges a label with a contour line drawn on the surface of the model of the workpiece, and generates a captured image of the label at the position where the model of the visual sensor is arranged based on imaging parameters set for the model of the visual sensor. The programming device detects the contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the model of the workpiece, and generates a program in which the industrial machine performs work on the workpiece with a tool based on the processing line and the normal vector.
As a result, according to this embodiment, it is possible to generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and to generate a program for an industrial machine to perform processing work of the trajectory based on the generated processing line.
The above is an outline of this embodiment.

Next, the configuration of this embodiment will be described in detail with reference to the drawings. Here, an example is shown in which models of a robot, tool, visual sensor, and workpiece as industrial machines are each placed in a virtual space, and a robot program is generated in which the robot performs work on the workpiece using the tool. Note that the present invention can also be applied to various types of industrial machines, such as machine tools, industrial robots, service robots, forging machines, laser processing machines, and injection molding machines.

FIG. 1 is a functional block diagram showing an example of the functional configuration of a programming device according to the first embodiment.
1, the programming device 1 is a known computer, and includes a control unit 10, an input unit 11, a display unit 12, and a storage unit 13. The control unit 10 includes a virtual space creation unit 101, a three-dimensional model placement unit 102, an image placement unit 103, an image capture unit 104, a contour line detection unit 105, a three-dimensional data conversion unit 106, a processing line generation unit 107, and a robot program generation unit 108.
The programming device 1 may be connected to a robot control device (not shown) that controls the operation of a robot (not shown) via a network (not shown) such as a LAN (Local Area Network) or the Internet. Alternatively, the programming device 1 may be directly connected to the robot control device (not shown) via a connection interface (not shown).

<Input unit 11>
The input unit 11 is, for example, a keyboard or a touch panel arranged on the display unit 12 (described later), and receives input from a user.

<Display unit 12>
The display unit 12 is, for example, a liquid crystal display, etc. As described later, the display unit 12 displays, for example, 3D CAD data or the like (hereinafter also referred to as a "robot model") that represents a robot (not shown) in three dimensions input (selected) by a user via the input unit 11, 3D CAD data or the like (hereinafter also referred to as a "tool model") that represents a tool (not shown) attached to the robot in three dimensions, 3D CAD data or the like (hereinafter also referred to as a "visual sensor model") that represents a visual sensor (not shown) in three dimensions, and 3D CAD data or the like (hereinafter also referred to as a "work model") that represents a work (not shown) in three dimensions.

<Storage unit 13>
The storage unit 13 is a solid state drive (SSD), a hard disk drive (HDD), etc. The storage unit 13 may store a generation program for generating a robot program in which a robot (not shown) performs an operation on a workpiece using a tool, a generated robot program, etc. The storage unit 13 also has a model storage unit 131.
As described above, the model storage unit 131 stores, for example, 3D CAD data (robot model) of a robot (not shown), 3D CAD data (tool model) of a tool (not shown), 3D CAD data (visual sensor model) of a visual sensor (not shown), and 3D CAD data (workpiece model) of a workpiece (not shown) that are input (selected) by a user via the input unit 11 and displayed on the display unit 12.

<Control Unit 10>
The control unit 10 has a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CMOS (Complementary Metal-Oxide-Semiconductor) memory, etc., which are configured to be able to communicate with each other via a bus, and are well known to those skilled in the art.
The CPU is a processor that controls the entire programming device 1. The CPU reads out the system program and application program stored in the ROM via the bus, and controls the entire programming device 1 according to the system program and application program. As a result, as shown in Fig. 1, the control unit 10 is configured to realize the functions of a virtual space creation unit 101, a three-dimensional model placement unit 102, an image placement unit 103, an image capture unit 104, a contour line detection unit 105, a three-dimensional data conversion unit 106, a processing line generation unit 107, and a robot program generation unit 108. The RAM stores various data such as temporary calculation data and display data. In addition, the CMOS memory is backed up by a battery (not shown) and is configured as a non-volatile memory that retains its stored state even when the power supply of the programming device 1 is turned off.

The virtual space creation unit 101 creates a virtual space that is a three-dimensional representation of a work space in which a robot (not shown), a tool (not shown), a visual sensor (not shown), and a workpiece (not shown) are arranged.

The three-dimensional model placement unit 102 places a robot model of a robot (not shown), a tool model of a tool (not shown), a visual sensor model of a visual sensor (not shown), and a work model of a work (not shown) within the three-dimensional virtual space created by the virtual space creation unit 101 in response to, for example, a user's input operation on the input unit 11.
FIG. 2 is a diagram showing an example of an industrial machinery system model in a virtual space.
2, in order to place a robot (not shown) in a virtual space, the three-dimensional model placement unit 102 reads out a robot model of the robot from the model storage unit 131. The three-dimensional model placement unit 102 places the read out robot model in the virtual space.
Furthermore, the three-dimensional model placement unit 102 reads out a tool model of a tool (not shown) from the model storage unit 131, and places the read tool model at the tool tip of the robot model of the robot in the virtual space.
Furthermore, the three-dimensional model placement unit 102 reads out a visual sensor model of a visual sensor (not shown) from the model storage unit 131, and places the visual sensor model of the read visual sensor at the tip of the arm of the robot model of the robot in the virtual space.
Furthermore, in order to place a workpiece model of a work (not shown) in the virtual space, the three-dimensional model placement unit 102 reads out the workpiece model of the work from the model storage unit 131. The three-dimensional model placement unit 102 places the read out workpiece model of the work in the virtual space.
In FIG. 2, the workpiece model is placed on a rectangular parallelepiped model of a jig or the like (not shown).

The image placement unit 103 places a label, such as a sticker, with an outline drawn on the surface of the work model, as shown in FIG. 3, for example. Note that the label shown in FIG. 3 has the outline of the letter "F" drawn on it.

As shown in Figure 4, the image capture unit 104 operates a robot model in a virtual space, positions the visual sensor model so that a vertical line to the center of the label indicated by the dotted line coincides with the optical axis of the visual sensor model, and adjusts the height of the visual sensor model so that the entire label falls within the field of view of the visual sensor model.
It is assumed that the visual sensor model is set with the imaging parameters necessary to capture images similar to those of a visual sensor in real space, such as the position of the sensor model coordinate system as viewed from the robot model coordinate system, focal length, image size, etc.
The image capturing unit 104 generates a pseudo captured image based on the imaging parameters set for the visual sensor model, which is a representation of the image captured by the visual sensor model capturing an image of the label at the position where the visual sensor model is placed. As shown in Fig. 4, the captured image shows the outline of the letter "F".

The contour line detection unit 105 detects contour lines by performing image processing on the pseudo captured image captured by the image capturing unit 104 .
Specifically, the contour detection unit 105 detects the black portion of the label in the pseudo captured image as a contour line by using, for example, a known method (for example, line drawing extraction, etc.).

The three-dimensional data conversion unit 106 converts the contours detected by the contour detection unit 105 into a set of three-dimensional data.
Specifically, for example, as shown in Fig. 5, the three-dimensional data conversion unit 106 converts the three-dimensional position coordinates (X, Y, Z) of three-dimensional points (black points in Fig. 5) on the detected contour line based on the robot model coordinate system or the sensor model coordinate system in the captured image captured by the visual sensor model into a set of three-dimensional data. The three-dimensional data conversion unit 106 stores the converted three-dimensional data in the storage unit 13.

The processing line generating unit 107 generates a processing line from a set of three-dimensional data.
Specifically, the processing line generating unit 107 generates a processing line indicated by a dashed line as shown in FIG. 6 by, for example, connecting each of the three-dimensional data of the contour lines stored in the storage unit 13 .
In addition, the processing line generation unit 107 calculates the surface of the work model on which the contour label is placed from the three-dimensional data of the contour stored as a collection of three-dimensional points, as shown in Figure 7, and calculates the normal vector to the surface of the work model at each point on the processing line.

The robot program generating unit 108 generates a program based on the processing lines and normal vectors generated by the processing line generating unit 107 .
Specifically, the robot program generating unit 108 calculates the posture of the robot model at each three-dimensional point on the processing line from the normal vector to the surface of the workpiece model at each three-dimensional point on the processing line, and sets teaching points of the robot program along the processing line, as shown in Fig. 8. The robot program generating unit 108 generates a robot program in which the robot model performs work on the workpiece model using the tool model based on the processing line and the normal vector of the processing line to the surface of the workpiece model.
The programming device 1 then executes a simulation of the generated robot program, allowing the user to confirm that the robot model moves the tool model along the machining line of the letter "F", as shown in FIG. 9.

<Program Generation Process of Programming Device 1>
Next, the flow of a program generation process of the programming device 1 will be described with reference to FIG.
10 is a flowchart illustrating the program generation process of the programming device 1. The flow shown here is executed every time an instruction to generate a robot program is received.

In step S1, the three-dimensional model placement unit 102 places a robot model of a robot (not shown), a tool model of a tool (not shown), a visual sensor model of a visual sensor (not shown), and a work model of a work (not shown) that constitute the industrial machinery system model in the three-dimensional virtual space created by the virtual space creation unit 101 in response to an input operation by the user of the input unit 11.

In step S2, the image placement unit 103 places a label with a contour line drawn on the surface of the work model.

In step S3, the image capturing unit 104 operates the robot model in the virtual space, positions the visual sensor model so that a vertical line to the center of the label on the work model coincides with the optical axis of the visual sensor model, and adjusts the height of the visual sensor model so that the entire label falls within the field of view of the visual sensor model. Based on the imaging parameters set for the visual sensor model, the image capturing unit 104 generates a pseudo captured image in which the label is captured by the visual sensor model at the position where the visual sensor model is placed.

In step S4, the contour detection unit 105 detects contours by performing image processing on the pseudo-image captured in step S3.

In step S5, the three-dimensional data conversion unit 106 converts the contours detected in step S4 into a collection of three-dimensional data.

In step S6, the processing line generation unit 107 generates a processing line from the collection of three-dimensional data.

In step S7, the processing line generation unit 107 calculates the surface of the work model on which the contour label is placed from the three-dimensional data of the contour stored as a collection of three-dimensional points, and calculates the normal vector to the surface of the work model at each point on the processing line.

In step S8, the robot program generation unit 108 generates a program based on the processing lines and normal vectors generated in step S7.

As described above, the programming device 1 according to the first embodiment places a label with a contour line on the surface of a workpiece model placed in a virtual space, and generates a captured image of the label at the position where the visual sensor model is placed based on the imaging parameters set for the visual sensor model. The programming device 1 detects the contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the workpiece model, and generates a program for an industrial machine to perform work on a workpiece with a tool based on the processing line and the normal vector. As a result, the programming device 1 can generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and generate a program for an industrial machine to perform a processing operation of the trajectory based on the generated processing line.
Furthermore, the programming device 1 does not require the operator to be highly skilled, and can reduce the effort and time required for teaching work.
The first embodiment has been described above.

Second Embodiment
Next, a second embodiment will be described. In the first embodiment, the programming device 1 places a label on the surface of the model of the workpiece, on which a contour line is drawn that does not intersect with another contour line, and generates a captured image of the label at the position where the model of the visual sensor is placed based on the imaging parameters set for the model of the visual sensor. The programming device 1 detects a contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the workpiece, and generates a program in which an industrial machine performs work on the workpiece with a tool based on the processing line and the normal vector. In contrast, in the second embodiment, the programming device 1A places a label on the surface of the model of the workpiece, on which a contour line is drawn that intersects with another contour line at least at one point, and performs image processing on the captured image in which the contour line is captured to detect the contour line and divide it into a plurality of line segments, and combines at least two line segments drawn consecutively among the divided plurality of line segments into one line segment, and specifies the order of work to be performed by the program for each line segment. This is different from the first embodiment.
As a result, the programming device 1A of the second embodiment can generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and generate a program for an industrial machine to perform processing work of the trajectory based on the generated processing line.
The second embodiment will now be described.

Fig. 11 is a functional block diagram showing an example of the functional configuration of a programming device according to the second embodiment. Elements having the same functions as those of the programming device 1 in Fig. 1 are given the same reference numerals and detailed description thereof will be omitted.
11, the programming device 1A according to the second embodiment has a control unit 10a, an input unit 11, a display unit 12, and a storage unit 13. The control unit 10a includes a virtual space creation unit 101, a three-dimensional model placement unit 102, an image placement unit 103, an image capturing unit 104, a contour line detection unit 105a, a three-dimensional data conversion unit 106a, a processing line generation unit 107a, and a robot program generation unit 108a.
The input unit 11, the display unit 12, and the storage unit 13 have the same functions as the input unit 11, the display unit 12, and the storage unit 13 in the first embodiment.
The model storage unit 131 has the same function as the model storage unit 131 in the first embodiment.

<Control unit 10a>
The control unit 10a includes a CPU, a ROM, a RAM, a CMOS memory, etc., which are configured to be able to communicate with each other via a bus, and are well known to those skilled in the art.
The CPU is a processor that controls the entire programming device 1A. The CPU reads out the system program and application program stored in the ROM via the bus, and controls the entire programming device 1A in accordance with the system program and application program. As a result, as shown in Fig. 11, the control unit 10a is configured to realize the functions of a virtual space creation unit 101, a three-dimensional model placement unit 102, an image placement unit 103, an image capture unit 104, a contour line detection unit 105a, a three-dimensional data conversion unit 106a, a processing line generation unit 107a, and a robot program generation unit 108a.
The virtual space creation unit 101, the three-dimensional model placement unit 102, the image placement unit 103, and the image capture unit 104 have functions equivalent to those of the virtual space creation unit 101, the three-dimensional model placement unit 102, the image placement unit 103, and the image capture unit 104 in the first embodiment.

The contour line detection unit 105a detects contour lines by performing image processing on the pseudo captured image captured by the image capturing unit 104.
In the following, we will use as an example a label with contour lines that intersect at least one point, as shown in Figure 12, in which a label with the letter "a" is placed on the work model.
13, the contour line detection unit 105a acquires a pseudo captured image from the image capturing unit 104. The pseudo captured image is generated by capturing an image of a label at a position where a visual sensor model is placed based on set imaging parameters, the visual sensor model being placed so that a vertical line to the center of the label indicated by a dashed dotted line coincides with the optical axis of the visual sensor model. The contour line detection unit 105a detects the black part of the label in the pseudo captured image as a contour line by using a known method (e.g., line drawing extraction, etc.) in the same way as the contour line detection unit 105 in the first embodiment.
As shown in FIG. 14, the contour detection unit 105a divides the detected contour of the character "a" into eleven line segments L1 to L11 based on the positions of the intersections.
The contour detection unit 105a combines at least two selected line segments from among the line segments L1 to L11 into one line segment in response to, for example, a user's input operation on the input unit 11. Specifically, as shown in Fig. 15, the contour detection unit 105a combines the line segments L1 and L2 into one line segment C1. The contour detection unit 105a also combines the line segments L3 to L6 into one line segment C2. The contour detection unit 105a also combines the line segments L7 to L11 into one line segment C3.
In response to an input operation of the input unit 11 by the user, the contour line detection unit 105a instructs the robot program to perform work in the order of line segments C1, C2, and C3, for example.

The three-dimensional data conversion unit 106a converts the contours detected by the contour detection unit 105a into a set of three-dimensional data, similar to the three-dimensional data conversion unit 106 of the first embodiment.
Specifically, for example, as shown in Fig. 16, the three-dimensional data conversion unit 106a converts the three-dimensional position coordinates (X, Y, Z) of three-dimensional points (black points in Fig. 16) on the detected contour line based on the robot model coordinate system or the sensor model coordinate system in the captured image captured by the visual sensor model into a set of three-dimensional data. The three-dimensional data conversion unit 106a stores the converted three-dimensional data in the storage unit 13.

The processing line generating unit 107a generates a processing line from a set of three-dimensional data, similarly to the processing line generating unit 107 of the first embodiment.
Specifically, the processing line generating unit 107a generates a processing line indicated by a dashed line as shown in FIG. 17, for example, by connecting each of the three-dimensional data of the contour lines stored in the storage unit 13.
In addition, the processing line generating unit 107a calculates the surface of the work model on which an image displaying the contour line is placed from the three-dimensional data of the contour line stored as a collection of three-dimensional points, as shown in Figure 18, and calculates the normal vector to the surface of the work model at each point on the processing line.

The robot program generating unit 108a generates a program based on the order of line segments specified by the contour line detecting unit 105a and the processing lines and normal vectors generated by the processing line generating unit 107a.
Specifically, the robot program generating unit 108a calculates the posture of the robot model at each three-dimensional point on the processing line from the normal vector to the surface of the workpiece model at each three-dimensional point on the processing line, and sets teaching points of the robot program along the processing line, as shown in Fig. 19. The robot program generating unit 108a generates a robot program in which the robot model performs work in the order of line segments C1 to C3 specified by the tool model with respect to the workpiece model, based on the processing line and the normal vector of the processing line to the surface of the workpiece model.

<Program Generation Process of Programming Device 1A>
Next, the flow of the program generation process of the programming device 1A will be described with reference to FIG.
20 is a flowchart illustrating a program generation process of the programming device 1 A. The flow shown here is executed every time an instruction to generate a robot program is received.
The processes in steps S1 to S4 and steps S6 to S8 are similar to those in steps S1 to S3 and steps S5 to S7 in FIG. 10, and therefore will not be described.

In step S4a, the contour detection unit 105a detects a contour by performing image processing on the pseudo image captured in step S3. If the detected contour has an intersection, the contour detection unit 105a divides it into multiple line segments based on the position of the intersection. In response to the user's input operation on the input unit 11, the contour detection unit 105a combines at least two selected line segments from the multiple line segments into one line segment, and specifies the order in which work is to be performed for each line segment by the robot program.

In step S8a, the robot program generation unit 108a generates a program based on the order of each line segment specified in step S4a and the processing lines and normal vectors generated in step S7.

As described above, when a label with a contour line that intersects with another contour line at at least one point is placed on the surface of a workpiece, the programming device 1A according to the second embodiment detects the contour line by performing image processing on a captured image of the contour line, divides the contour line into a plurality of line segments, combines at least two of the divided plurality of line segments that are drawn consecutively into one line segment, and specifies the order of work to be performed for each line segment by a program. In this way, the programming device 1A can generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and generate a program for an industrial machine to perform a processing operation of the trajectory based on the generated processing line.
Furthermore, the programming device 1A does not require the operator to be skilled, and can reduce the effort and time required for teaching work.
The second embodiment has been described above.

Third Embodiment
Next, a third embodiment will be described. In the first embodiment, the programming device 1 places a label on the surface of the model of the workpiece, on which a contour line is drawn that does not intersect with another contour line, and generates a captured image of the label at the position where the model of the visual sensor is placed based on the imaging parameters set for the model of the visual sensor. The programming device 1 detects a contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the model of the workpiece, and generates a program for an industrial machine to perform work on the workpiece with a tool based on the processing line and the normal vector. In the second embodiment, the programming device 1A differs from the first embodiment in that it places a label on the surface of the model of the workpiece, on which a contour line is drawn that intersects with another contour line at least at one point, detects the contour line by performing image processing on the captured image in which the contour line is captured, divides the contour line into a plurality of line segments, combines at least two line segments drawn consecutively among the divided plurality of line segments into one line segment, and specifies the order of work to be performed by the program for each line segment. In contrast, in the third embodiment, the programming device 1B differs from the first and second embodiments in that, in an industrial machine, a tool attached to the industrial machine, a visual sensor, and a workpiece arranged in real space, a label having a contour line drawn on the surface of the workpiece is arranged, and an image of the label captured at the position where the visual sensor is arranged is obtained from the visual sensor based on imaging parameters set in the visual sensor.
As a result, the programming device 1B of the third embodiment can generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and generate a program for an industrial machine to perform processing work of the trajectory based on the generated processing line.
The third embodiment will now be described.

Fig. 21 is a functional block diagram showing an example of the functional configuration of a programming device according to the third embodiment. Elements having the same functions as those of the programming device 1 in Fig. 1 are given the same reference numerals and detailed description thereof will be omitted.
21, a programming device 1B according to the third embodiment has a control unit 10b, an input unit 11, a display unit 12, and a storage unit 13b. The control unit 10b includes an image arrangement unit 103b, an image capturing unit 104b, a contour line detection unit 105, a three-dimensional data conversion unit 106, a processing line generation unit 107, and a robot program generation unit 108.
The input unit 11 and the display unit 12 have the same functions as the input unit 11 and the display unit 12 in the first embodiment.

<Storage unit 13b>
The memory unit 13b is an SSD, HDD, etc., similar to the memory unit 13 in the first embodiment, and may store a generation program that generates a robot program in which a robot (not shown) performs work on a workpiece using a tool, the generated robot program, etc.

<Control unit 10b>
The control unit 10b includes a CPU, a ROM, a RAM, a CMOS memory, etc., which are configured to be able to communicate with each other via a bus, and are well known to those skilled in the art.
The CPU is a processor that controls the entire programming device 1B. The CPU reads out the system program and application program stored in the ROM via the bus, and controls the entire programming device 1B in accordance with the system program and application program. As a result, as shown in Fig. 21, the control unit 10b is configured to realize the functions of an image arrangement unit 103b, an image pickup unit 104b, a contour line detection unit 105, a three-dimensional data conversion unit 106, a processing line generation unit 107, and a robot program generation unit 108.
In addition, the contour line detection unit 105, the three-dimensional data conversion unit 106, the processing line generation unit 107, and the robot program generation unit 108 have functions equivalent to the contour line detection unit 105, the three-dimensional data conversion unit 106, the processing line generation unit 107, and the robot program generation unit 108 in the first embodiment.

The image placement unit 103b may, for example, use a projector (not shown) to project and place a label with a contour line drawn on the surface of the workpiece in the same manner as in FIG. 3.

The image capturing unit 104b operates the robot in real space via a robot control device not shown, and, as in the case of Figure 4, positions the visual sensor so that a vertical line to the center of the label indicated by the dotted line coincides with the optical axis of the visual sensor, and adjusts the height of the visual sensor so that the entire label falls within the field of view of the visual sensor.
The image capturing unit 104b captures an image of the label at the position where the visual sensor is disposed based on the imaging parameters set for the visual sensor, and outputs the captured image to the contour line detection unit 105.

<Program Generation Process of Programming Device 1B>
Next, the flow of the program generation process of the programming device 1B will be described with reference to FIG.
22 is a flowchart illustrating a program generation process of the programming device 1B. The flow shown here is executed every time an instruction to generate a robot program is received.
The processes in steps S33 to S37 are similar to those in steps S4 to S8 in FIG. 10, and therefore will not be described.

In step S31, the image placement unit 103b uses a projector (not shown) or the like to project and place a label with a contour line drawn on the surface of the work.

In step S32, the image capturing unit 104b operates the robot in real space to position the visual sensor so that a vertical line coincides with the optical axis of the visual sensor at the center of the label on the workpiece, and adjusts the height of the visual sensor so that the entire label falls within the field of view of the visual sensor. The image capturing unit 104b obtains an image of the label at the position where the visual sensor is positioned, based on the imaging parameters set for the visual sensor.

As described above, the programming device 1B according to the third embodiment places a label with a contour drawn on the surface of a workpiece, and acquires an image of the label at the position where the visual sensor is placed from the visual sensor based on the imaging parameters set in the visual sensor. The programming device 1B detects a contour by performing image processing on the captured image, generates a processing line based on the detected contour, calculates a normal vector of the generated processing line with respect to the surface of the workpiece, and generates a program for an industrial machine to perform work on the workpiece with a tool based on the processing line and the normal vector. As a result, the programming device 1B can generate a processing line that draws an arbitrary and complex trajectory on the surface of the workpiece, and generate a program for an industrial machine to perform a processing work of the trajectory based on the generated processing line.
Furthermore, the programming device 1B does not require the operator to be highly skilled, and can reduce the effort and time required for teaching work.
The third embodiment has been described above.

<Modification of the third embodiment>
In the third embodiment, the label to be placed on the workpiece is one in which the contour lines of the letter "F" and the like do not intersect with each other, but this is not limited thereto. For example, the label may be one in which the contour lines of the letter "a" and the like intersect with each other at at least one point. In this case, similar to the second embodiment, the programming device 1B may detect the contour lines by performing image processing on the captured image in which the contour lines are captured, divide the contour lines into a plurality of line segments, combine at least two line segments drawn consecutively among the divided plurality of line segments into one line segment, and specify the order of operations to be performed by the program for each line segment.

As described above in the first to third embodiments, the

programming devices

1, 1A, and 1B disclosed herein can generate machining lines that trace arbitrary and complex trajectories on the surface of a workpiece, and generate a program for an industrial machine to perform machining work of the trajectory based on the generated machining lines.

<Modification 1>
In the above-described first to third embodiments, the surface of the workpiece (workpiece model) is a flat surface, but is not limited thereto. The surface of the workpiece (workpiece model) may have any shape, such as unevenness. In this case, a sticker with a label drawn on it may be attached to the surface of the workpiece (workpiece model) of any shape, or a label may be projected by a projector or the like.

<Modification 2>
In addition, for example, in the above-described embodiment, the

programming devices

1, 1A, and 1B generate robot programs for a robot as an industrial machine, but are not limited to this. For example, the

programming devices

1, 1A, and 1B may generate programs for a machine tool, a forging machine, a laser processing machine, an injection molding machine, or the like as an industrial machine.

<Modification 3>
Also, for example, in the above-mentioned embodiment, the visual sensor (visual sensor model) is attached to the tip of the arm of the robot (robot model), but is not limited to this. For example, the visual sensor (visual sensor model) may be fixed and placed at a position on a wall, ceiling, etc., separate from the robot (robot model) so that the entire label placed on the work fits within the field of view, as shown in Fig. 23 .

In addition, each function included in the

programming devices

1, 1A, and 1B in the first to third embodiments can be realized by hardware, software, or a combination of these. Here, being realized by software means being realized by a computer reading and executing a program.

The program can be stored and provided to the computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). The program may also be provided to the computer by various types of temporary computer readable media. Examples of temporary computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer readable medium can provide the program to the computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

The step of writing the program to be recorded on the recording medium includes not only processes that are performed chronologically according to the order, but also processes that are not necessarily performed chronologically but are executed in parallel or individually. The step of writing the program may also be performed by cloud computing.

Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the contents described in the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-mentioned embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values or formulas are used to explain the above-mentioned embodiments.

The following supplementary notes are further disclosed regarding the above-described embodiment and modified examples.
(Appendix 1)
A programming device (1) is a programming device that places an industrial machine, a tool attached to the industrial machine, a visual sensor, and a work model within a virtual space, and generates a program in which the industrial machine model performs an operation on a work model using a tool model. The programming device places a label with a contour line on the surface of the work model, generates an image of the label at the position where the visual sensor model is placed based on imaging parameters set for the visual sensor model, detects the contour line by applying image processing to the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the work model, and generates a program based on the processing line and the normal vector.
(Appendix 2)
The programming device (1B) is a programming device that generates a program in which an industrial machine performs an operation on a workpiece using a tool, for an industrial machine, a tool attached to the industrial machine, a visual sensor, and a workpiece that are arranged in real space, and the programming device places a label with a contour line drawn on the surface of the workpiece, obtains an image of the label captured at the position where the visual sensor is arranged based on imaging parameters set in the visual sensor, detects the contour line by performing image processing on the captured image, generates a processing line based on the detected contour line, calculates a normal vector of the generated processing line with respect to the surface of the workpiece, and generates a program based on the processing line and the normal vector.
(Appendix 3)
In the programming device (1A) described in Supplementary Note 1 or Supplementary Note 2, a contour line is divided into a plurality of line segments, and each of the plurality of line segments is converted into a set of three-dimensional data.
(Appendix 4)
In the programming device (1A) described in Appendix 3, at least two consecutive line segments are selected from a plurality of line segments and joined into one line segment.
(Appendix 5)
In the programming device (1A) described in Appendix 3, the order in which operations are performed on a plurality of line segments is designated by a program.
(Appendix 6)
In the programming device (1) described in Appendix 1, the visual sensor is placed at a position where the entire label falls within the field of view of the model of the visual sensor.
(Appendix 7)
In the programming device (1B) described in Appendix 2, the visual sensor is placed in a position where the entire label fits within the field of view of the visual sensor.

Reference Signs List

1, 1A, 1B Programming device 10, 10a, 10b Control unit 101 Virtual space generation unit 102 Three-dimensional

model arrangement unit

103, 103b

Image arrangement unit

104, 104b Image capture unit 105, 105a Contour line detection unit 106, 106a Three-dimensional data conversion unit 107, 107a Processing line generation unit 108, 108a Robot program generation unit 11 Input unit 12

Display unit

13, 13b Storage unit 131 Model storage unit

Claims

A programming device that arranges models of an industrial machine, a tool attached to the industrial machine, a visual sensor, and a workpiece in a virtual space, and generates a program in which the model of the industrial machine performs an operation on the model of the workpiece using the model of the tool,
Placing a label having a contour line on a surface of the model of the workpiece;
generating a captured image of the label at a position where the model of the visual sensor is disposed based on imaging parameters set for the model of the visual sensor;
Detecting the contour line by performing image processing on the captured image;
A processing line is generated based on the detected contour line, and a normal vector of the generated processing line with respect to a surface of the model of the workpiece is calculated;
A programming device that generates the program based on the processing line and the normal vector.
A programming device that generates a program for an industrial machine that performs an operation on a workpiece using a tool attached to the industrial machine, a visual sensor, and a workpiece that are arranged in a real space, the programming device comprising:
Placing a label having a contour line on the surface of the workpiece;
acquiring, from the visual sensor, an image of the label captured at a position where the visual sensor is disposed, based on imaging parameters set for the visual sensor;
Detecting the contour line by performing image processing on the captured image;
A processing line is generated based on the detected contour line, and a normal vector of the generated processing line with respect to a surface of the workpiece is calculated;
A programming device that generates the program based on the processing line and the normal vector.
The programming device according to claim 1 or 2, which divides the contour line into a plurality of line segments and converts each of the plurality of line segments into a set of three-dimensional data.
The programming device according to claim 3, which selects at least two consecutive line segments from the plurality of line segments and combines them into one line segment.
The programming device according to claim 3, which specifies the order in which the program performs the operations on the multiple line segments.
The programming device of claim 1, wherein the model of the visual sensor is positioned so that the entire label is within the field of view of the visual sensor.
The programming device according to claim 2, wherein the visual sensor is positioned so that the entire label is within the visual field of the visual sensor.